Jul 15 14:31:34.190133 ip-10-0-139-83 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 15 14:31:34.190148 ip-10-0-139-83 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 15 14:31:34.190157 ip-10-0-139-83 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 15 14:31:34.190476 ip-10-0-139-83 systemd[1]: Failed to start Kubernetes Kubelet. Jul 15 14:31:44.308001 ip-10-0-139-83 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 15 14:31:44.308022 ip-10-0-139-83 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 80f1ed6a61594e96a601e2f1096e94f2 -- Jul 15 14:33:47.182193 ip-10-0-139-83 systemd[1]: Starting Kubernetes Kubelet... Jul 15 14:33:48.743139 ip-10-0-139-83 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 14:33:48.743139 ip-10-0-139-83 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 14:33:48.743139 ip-10-0-139-83 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 14:33:48.743139 ip-10-0-139-83 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 14:33:48.743139 ip-10-0-139-83 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 14:33:48.744756 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.744654 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 14:33:48.748818 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748795 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 14:33:48.748818 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748814 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 14:33:48.748818 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748820 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 14:33:48.748818 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748824 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748830 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748836 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748843 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748850 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748855 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748859 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748864 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748868 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748872 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748876 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748880 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748884 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748888 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748892 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748897 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748901 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748905 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748909 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748913 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 14:33:48.749120 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748918 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748923 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748927 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748931 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748936 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748940 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748944 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748948 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748952 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748975 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748980 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748984 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748989 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748994 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.748999 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749005 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749010 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749015 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749019 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749024 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 14:33:48.749979 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749028 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749035 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749041 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749046 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749051 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749056 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749060 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749065 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749070 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749074 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749079 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749086 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749092 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749097 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749102 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749108 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749112 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749116 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749121 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 14:33:48.750848 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749125 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749129 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749134 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749140 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749144 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749151 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749156 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749161 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749165 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749172 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749177 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749181 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749186 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749190 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749195 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749199 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749204 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749208 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749214 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749218 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 14:33:48.751696 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749223 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749228 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749232 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749236 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749241 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749245 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749250 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749264 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749268 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749272 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749277 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749281 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749286 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749290 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749295 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749300 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.749306 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750699 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750711 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750716 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 14:33:48.752581 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750722 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750728 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750735 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750740 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750745 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750751 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750756 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750760 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750765 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750769 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750774 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750779 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750784 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750788 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750793 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750797 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750801 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750805 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750810 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750815 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 14:33:48.753527 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750820 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750825 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750829 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750834 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750838 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750843 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750847 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750852 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750859 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750866 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750873 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750878 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750883 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750888 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750893 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750901 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750905 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750910 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750914 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 14:33:48.754329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750919 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750924 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750929 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750933 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750938 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750943 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750948 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750952 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750956 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750979 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750984 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750988 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750993 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.750997 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751002 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751007 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751011 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751015 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751020 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751024 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 14:33:48.754938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751030 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751034 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751039 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751044 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751049 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751054 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751059 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751064 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751069 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751075 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751080 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751084 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751094 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751100 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751105 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751110 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751116 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751121 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751126 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 14:33:48.755532 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751131 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751135 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751140 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751144 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751149 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751153 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751158 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751163 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751167 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751172 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751177 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751181 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751186 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751191 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751195 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751201 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751208 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751214 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751862 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751870 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 14:33:48.756255 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751875 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751879 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751886 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751891 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751896 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751900 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751905 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751909 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751914 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751918 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751923 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751927 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751931 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751936 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751940 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751945 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751949 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751955 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751980 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751984 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 14:33:48.756765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751989 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751994 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.751998 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752003 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752008 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752012 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752017 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752021 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752026 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752031 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752037 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752044 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752049 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752053 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752058 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752063 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752068 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752074 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752078 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752083 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 14:33:48.757308 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752088 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752092 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752097 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752101 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752106 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752114 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752119 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752124 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752129 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752134 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752139 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752143 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752147 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752152 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752156 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752160 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752167 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752174 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752179 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 14:33:48.757805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752184 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752190 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752195 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752200 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752205 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752211 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752216 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752221 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752226 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752234 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752240 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752245 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752250 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752256 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752261 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752265 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752270 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752275 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752281 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 14:33:48.758320 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752288 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752292 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752296 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752301 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752306 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752310 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752314 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752319 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752324 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752328 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752332 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752337 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752341 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752346 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752350 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752355 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752359 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752363 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.752368 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752475 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752487 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 14:33:48.758805 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752496 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752505 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752513 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752519 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752526 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752534 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752539 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752545 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752551 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752557 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752565 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752570 2574 flags.go:64] FLAG: --cgroup-root="" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752575 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752580 2574 flags.go:64] FLAG: --client-ca-file="" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752584 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752589 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752596 2574 flags.go:64] FLAG: --cluster-domain="" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752601 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752606 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752612 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752618 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752625 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752631 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 14:33:48.759424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752636 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752642 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752647 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752652 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752657 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752663 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752669 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752676 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752681 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752686 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752691 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752696 2574 flags.go:64] FLAG: --enable-server="true" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752703 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752710 2574 flags.go:64] FLAG: --event-burst="100" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752715 2574 flags.go:64] FLAG: --event-qps="50" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752720 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752725 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752731 2574 flags.go:64] FLAG: --eviction-hard="" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752737 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752742 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752748 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752753 2574 flags.go:64] FLAG: --eviction-soft="" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752758 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752763 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752768 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 14:33:48.760021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752773 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752777 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752782 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752788 2574 flags.go:64] FLAG: --feature-gates="" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752795 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752800 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752806 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752812 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752817 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752822 2574 flags.go:64] FLAG: --help="false" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752827 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-139-83.ec2.internal" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752832 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752838 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752843 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752848 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752855 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752863 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752868 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752873 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752878 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752884 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752890 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752895 2574 flags.go:64] FLAG: --kube-reserved="" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752900 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 14:33:48.760699 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752905 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752910 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752916 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752921 2574 flags.go:64] FLAG: --lock-file="" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752926 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752931 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752937 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752946 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752952 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752978 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752984 2574 flags.go:64] FLAG: --logging-format="text" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752989 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.752995 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753000 2574 flags.go:64] FLAG: --manifest-url="" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753006 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753013 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753019 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753026 2574 flags.go:64] FLAG: --max-pods="110" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753031 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753037 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753042 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753047 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753052 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753057 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753064 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 14:33:48.761370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753077 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753083 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753088 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753094 2574 flags.go:64] FLAG: --pod-cidr="" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753100 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753110 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753115 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753121 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753126 2574 flags.go:64] FLAG: --port="10250" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753131 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753136 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-061a556bf4d83f668" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753142 2574 flags.go:64] FLAG: --qos-reserved="" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753147 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753152 2574 flags.go:64] FLAG: --register-node="true" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753158 2574 flags.go:64] FLAG: --register-with-taints="" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753164 2574 flags.go:64] FLAG: --registry-burst="10" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753169 2574 flags.go:64] FLAG: --registry-qps="5" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753174 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753179 2574 flags.go:64] FLAG: --reserved-memory="" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753185 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753190 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753196 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753201 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753206 2574 flags.go:64] FLAG: --runonce="false" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753212 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 14:33:48.762158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753217 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753222 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753227 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753231 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753237 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753242 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753253 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753258 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753263 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753268 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753273 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753278 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753285 2574 flags.go:64] FLAG: --system-cgroups="" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753291 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753300 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753305 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753310 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753317 2574 flags.go:64] FLAG: --tls-min-version="" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753321 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753327 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753332 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753337 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753342 2574 flags.go:64] FLAG: --v="2" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753349 2574 flags.go:64] FLAG: --version="false" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753355 2574 flags.go:64] FLAG: --vmodule="" Jul 15 14:33:48.763165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753361 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.753367 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753560 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753568 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753573 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753577 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753582 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753596 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753601 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753605 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753610 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753614 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753619 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753625 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753630 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753634 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753639 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753643 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753648 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753652 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 14:33:48.763814 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753659 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753664 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753669 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753675 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753679 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753684 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753691 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753696 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753702 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753706 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753711 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753716 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753720 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753725 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753730 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753734 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753739 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753743 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753748 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753753 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 14:33:48.764353 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753758 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753762 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753767 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753772 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753776 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753782 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753787 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753791 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753796 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753801 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753806 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753810 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753818 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753824 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753828 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753833 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753838 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753843 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753848 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753852 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 14:33:48.764857 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753857 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753861 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753866 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753870 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753875 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753879 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753884 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753888 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753893 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753900 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753906 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753911 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753916 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753920 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753925 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753930 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753935 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753940 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753945 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 14:33:48.765443 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753950 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753955 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753977 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753982 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753987 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.753993 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754000 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754007 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754013 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754018 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754023 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754028 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754033 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754038 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754043 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754048 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754053 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754058 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754062 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 14:33:48.765935 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754067 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 14:33:48.766471 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754072 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 14:33:48.766471 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.754076 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 14:33:48.766471 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.754646 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 14:33:48.766471 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.763924 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 14:33:48.766471 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.763955 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 14:33:48.766471 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.763978 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 14:33:48.766471 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.763984 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 14:33:48.766471 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764045 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 14:33:48.766471 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764052 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 14:33:48.766471 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764057 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 14:33:48.766471 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764061 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 14:33:48.766471 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764064 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 14:33:48.766471 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764068 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 14:33:48.766471 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764071 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 14:33:48.766471 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764074 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764077 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764080 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764083 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764085 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764088 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764091 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764096 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764100 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764103 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764106 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764110 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764113 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764116 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764120 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764123 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764125 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764129 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764131 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 14:33:48.766883 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764134 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764137 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764140 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764143 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764146 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764149 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764152 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764155 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764158 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764161 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764164 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764166 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764169 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764172 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764175 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764177 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764180 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764183 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764186 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764189 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 14:33:48.767398 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764192 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764195 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764198 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764201 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764203 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764206 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764209 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764212 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764214 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764217 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764219 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764222 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764225 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764227 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764230 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764234 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764237 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764240 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764242 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764245 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 14:33:48.767902 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764248 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764251 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764254 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764257 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764259 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764262 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764265 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764268 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764271 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764273 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764276 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764280 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764284 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764288 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764291 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764294 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764297 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764299 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764302 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 14:33:48.768435 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764305 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 14:33:48.768918 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764308 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 14:33:48.768918 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764311 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 14:33:48.768918 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764313 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 14:33:48.768918 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764316 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 14:33:48.768918 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764319 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 14:33:48.768918 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764321 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 14:33:48.768918 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764325 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 14:33:48.768918 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764328 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 14:33:48.768918 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764331 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 14:33:48.768918 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764333 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 14:33:48.768918 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764336 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 14:33:48.768918 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764339 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 14:33:48.768918 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764342 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 14:33:48.768918 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.764347 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 14:33:48.768918 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764471 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764477 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764480 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764484 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764487 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764490 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764493 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764496 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764499 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764502 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764504 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764507 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764511 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764514 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764516 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764519 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764522 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764525 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764528 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 14:33:48.769327 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764532 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764536 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764538 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764542 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764545 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764547 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764551 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764554 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764556 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764559 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764562 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764565 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764568 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764571 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764573 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764576 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764579 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764581 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764584 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764587 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 14:33:48.769812 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764590 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764592 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764595 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764598 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764601 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764603 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764607 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764610 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764612 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764615 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764618 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764621 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764624 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764627 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764629 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764632 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764635 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764637 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764640 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764644 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 14:33:48.770329 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764646 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764649 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764652 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764655 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764659 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764663 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764666 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764669 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764672 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764675 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764678 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764681 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764684 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764687 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764690 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764693 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764695 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764698 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764701 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764704 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 14:33:48.770835 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764707 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764710 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764713 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764716 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764719 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764723 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764726 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764728 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764731 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764734 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764737 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764740 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764743 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764745 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764748 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764751 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764753 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764756 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764759 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 14:33:48.771446 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:48.764762 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 14:33:48.771928 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.764767 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 14:33:48.771928 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.764883 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 14:33:48.771928 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.767531 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 14:33:48.771928 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.768914 2574 server.go:1019] "Starting client certificate rotation" Jul 15 14:33:48.771928 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.769032 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 14:33:48.771928 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.769076 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 14:33:48.771928 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.771728 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 14:33:48.773098 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.773079 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 14:33:48.787117 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.787094 2574 log.go:25] "Validated CRI v1 runtime API" Jul 15 14:33:48.790093 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.790079 2574 log.go:25] "Validated CRI v1 image API" Jul 15 14:33:48.791978 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.791954 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 14:33:48.794694 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.794668 2574 fs.go:135] Filesystem UUIDs: map[1babb617-65bd-40fd-9742-cb654c0f3819:/dev/nvme0n1p3 51f2cb3f-a2c5-4a88-b5b8-39f0d764a683:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 15 14:33:48.794764 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.794695 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 14:33:48.801611 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.801479 2574 manager.go:216] Machine: {Timestamp:2026-07-15 14:33:48.799680986 +0000 UTC m=+1.505757087 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090163 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b2e2c0ca87d3d8df421b0a725b734 SystemUUID:ec2b2e2c-0ca8-7d3d-8df4-21b0a725b734 BootID:80f1ed6a-6159-4e96-a601-e2f1096e94f2 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e7:34:35:a0:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e7:34:35:a0:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:ee:da:af:ec:59 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 14:33:48.801611 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.801598 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 14:33:48.801745 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.801734 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 14:33:48.803969 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.803898 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 14:33:48.804112 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.803966 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-83.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 14:33:48.804380 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.804364 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 14:33:48.804909 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.804897 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 14:33:48.804950 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.804911 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 14:33:48.804950 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.804919 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 14:33:48.805633 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.805622 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 14:33:48.807571 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.807559 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 14:33:48.811449 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.811437 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 15 14:33:48.811581 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.811571 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 14:33:48.811614 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.811602 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 15 14:33:48.811614 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.811613 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 14:33:48.811679 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.811627 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 14:33:48.811679 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.811637 2574 kubelet.go:398] "Adding apiserver pod source" Jul 15 14:33:48.811679 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.811647 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 14:33:48.812069 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.812057 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 14:33:48.812113 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.812077 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 14:33:48.814417 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.814402 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 14:33:48.816624 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.816609 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 14:33:48.816696 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.816642 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 14:33:48.816847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.816836 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 14:33:48.816881 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.816852 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 14:33:48.816881 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.816859 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 14:33:48.816881 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.816865 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 14:33:48.816992 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.816874 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 14:33:48.816992 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.816908 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 14:33:48.817328 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.817318 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 14:33:48.817359 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.817328 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 14:33:48.817359 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.817336 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 14:33:48.817359 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.817341 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 14:33:48.817359 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.817350 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 14:33:48.817359 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.817359 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 14:33:48.818101 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.818090 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 14:33:48.818101 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.818101 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 14:33:48.821611 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.821595 2574 server.go:1268] "Started kubelet" Jul 15 14:33:48.821697 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.821615 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-83.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 14:33:48.821756 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:48.821717 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 14:33:48.821756 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:48.821725 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-83.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 14:33:48.821852 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.821820 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 14:33:48.821852 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.821812 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 14:33:48.821981 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.821878 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 14:33:48.822695 ip-10-0-139-83 systemd[1]: Started Kubernetes Kubelet. Jul 15 14:33:48.824165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.824145 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 14:33:48.824649 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.824635 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 15 14:33:48.829935 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:48.829911 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 14:33:48.831226 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:48.830154 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-83.ec2.internal.18c27d2ba6701d58 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-83.ec2.internal,UID:ip-10-0-139-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-83.ec2.internal,},FirstTimestamp:2026-07-15 14:33:48.821568856 +0000 UTC m=+1.527644954,LastTimestamp:2026-07-15 14:33:48.821568856 +0000 UTC m=+1.527644954,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-83.ec2.internal,}" Jul 15 14:33:48.832080 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.832061 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 14:33:48.832153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.832089 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 14:33:48.832511 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:48.832422 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-83.ec2.internal\" not found" Jul 15 14:33:48.833124 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.833106 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 14:33:48.833124 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.833127 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 14:33:48.833269 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.833241 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 15 14:33:48.833269 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.833250 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 15 14:33:48.833368 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.833297 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 14:33:48.833420 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.833394 2574 factory.go:55] Registering systemd factory Jul 15 14:33:48.833476 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.833422 2574 factory.go:223] Registration of the systemd container factory successfully Jul 15 14:33:48.833552 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.833530 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vwwf2" Jul 15 14:33:48.833653 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.833623 2574 factory.go:153] Registering CRI-O factory Jul 15 14:33:48.833653 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.833636 2574 factory.go:223] Registration of the crio container factory successfully Jul 15 14:33:48.833763 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.833693 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 14:33:48.833763 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.833717 2574 factory.go:103] Registering Raw factory Jul 15 14:33:48.833763 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.833735 2574 manager.go:1196] Started watching for new ooms in manager Jul 15 14:33:48.834128 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.834114 2574 manager.go:349] Starting recovery of all containers Jul 15 14:33:48.842021 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:48.841824 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-83.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 14:33:48.842021 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:48.841861 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 14:33:48.843646 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.843603 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 14:33:48.845922 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.845898 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vwwf2" Jul 15 14:33:48.847490 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.847469 2574 manager.go:354] Recovery completed Jul 15 14:33:48.852464 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.852448 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 14:33:48.854623 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.854606 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasSufficientMemory" Jul 15 14:33:48.854705 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.854636 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 14:33:48.854705 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.854647 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasSufficientPID" Jul 15 14:33:48.855252 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.855239 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 14:33:48.855252 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.855252 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 14:33:48.855385 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.855270 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 15 14:33:48.855385 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.855252 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 14:33:48.856072 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.856050 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasSufficientMemory" Jul 15 14:33:48.856169 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.856087 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 14:33:48.856169 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.856102 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasSufficientPID" Jul 15 14:33:48.856382 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.856368 2574 policy_none.go:49] "None policy: Start" Jul 15 14:33:48.856461 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.856390 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 14:33:48.856461 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.856403 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 14:33:48.857476 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:48.857401 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-83.ec2.internal.18c27d2ba86872fa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-83.ec2.internal,UID:ip-10-0-139-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-83.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-83.ec2.internal,},FirstTimestamp:2026-07-15 14:33:48.854620922 +0000 UTC m=+1.560697020,LastTimestamp:2026-07-15 14:33:48.854620922 +0000 UTC m=+1.560697020,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-83.ec2.internal,}" Jul 15 14:33:48.857565 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.857494 2574 policy_none.go:47] "Start" Jul 15 14:33:48.891585 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.891567 2574 manager.go:337] "Starting Device Plugin manager" Jul 15 14:33:48.891682 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:48.891603 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 14:33:48.891682 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.891614 2574 server.go:87] "Starting device plugin registration server" Jul 15 14:33:48.891875 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.891862 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 14:33:48.891909 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.891880 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 14:33:48.892001 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.891985 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 14:33:48.892094 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.892076 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 14:33:48.892094 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.892087 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 14:33:48.892659 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:48.892641 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 14:33:48.892721 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:48.892689 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-83.ec2.internal\" not found" Jul 15 14:33:48.992703 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.992666 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 14:33:48.994447 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.993852 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasSufficientMemory" Jul 15 14:33:48.994447 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.993883 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 14:33:48.994447 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.993911 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasSufficientPID" Jul 15 14:33:48.994447 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:48.993941 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.006929 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.006909 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.007017 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:49.006941 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-83.ec2.internal\": node \"ip-10-0-139-83.ec2.internal\" not found" Jul 15 14:33:49.076953 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.076915 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 14:33:49.077120 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.076976 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 14:33:49.077120 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.077003 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 14:33:49.077120 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:49.077044 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 14:33:49.084587 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.084568 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 14:33:49.087843 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:49.087818 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-83.ec2.internal\" not found" Jul 15 14:33:49.180187 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.180139 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-83.ec2.internal"] Jul 15 14:33:49.180355 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.180255 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 14:33:49.181317 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.181298 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasSufficientMemory" Jul 15 14:33:49.181413 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.181341 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 14:33:49.181413 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.181352 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasSufficientPID" Jul 15 14:33:49.182844 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.182830 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 14:33:49.183043 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.183019 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.183148 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.183053 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 14:33:49.183727 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.183712 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasSufficientMemory" Jul 15 14:33:49.183810 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.183735 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 14:33:49.183810 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.183737 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasSufficientMemory" Jul 15 14:33:49.183810 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.183769 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 14:33:49.183810 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.183780 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasSufficientPID" Jul 15 14:33:49.183810 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.183750 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasSufficientPID" Jul 15 14:33:49.185233 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.185215 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.185332 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.185242 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 14:33:49.185935 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.185921 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasSufficientMemory" Jul 15 14:33:49.186042 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.185943 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 14:33:49.186042 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.185956 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeHasSufficientPID" Jul 15 14:33:49.188540 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:49.188525 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-83.ec2.internal\" not found" Jul 15 14:33:49.209768 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:49.209737 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-83.ec2.internal\" not found" node="ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.214462 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:49.214442 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-83.ec2.internal\" not found" node="ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.236203 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.236179 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eeee09bf5e958e41488ee1c3e17d2133-config\") pod \"kube-apiserver-proxy-ip-10-0-139-83.ec2.internal\" (UID: \"eeee09bf5e958e41488ee1c3e17d2133\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.236288 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.236216 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a8ba0b7cefb501e9fe51d83336aec9d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal\" (UID: \"a8ba0b7cefb501e9fe51d83336aec9d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.236288 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.236238 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8ba0b7cefb501e9fe51d83336aec9d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal\" (UID: \"a8ba0b7cefb501e9fe51d83336aec9d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.289755 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:49.289684 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-83.ec2.internal\" not found" Jul 15 14:33:49.337499 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.337467 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8ba0b7cefb501e9fe51d83336aec9d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal\" (UID: \"a8ba0b7cefb501e9fe51d83336aec9d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.337576 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.337503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eeee09bf5e958e41488ee1c3e17d2133-config\") pod \"kube-apiserver-proxy-ip-10-0-139-83.ec2.internal\" (UID: \"eeee09bf5e958e41488ee1c3e17d2133\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.337576 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.337531 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a8ba0b7cefb501e9fe51d83336aec9d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal\" (UID: \"a8ba0b7cefb501e9fe51d83336aec9d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.337646 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.337571 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8ba0b7cefb501e9fe51d83336aec9d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal\" (UID: \"a8ba0b7cefb501e9fe51d83336aec9d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.337646 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.337588 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a8ba0b7cefb501e9fe51d83336aec9d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal\" (UID: \"a8ba0b7cefb501e9fe51d83336aec9d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.337646 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.337574 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eeee09bf5e958e41488ee1c3e17d2133-config\") pod \"kube-apiserver-proxy-ip-10-0-139-83.ec2.internal\" (UID: \"eeee09bf5e958e41488ee1c3e17d2133\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.389874 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:49.389842 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-83.ec2.internal\" not found" Jul 15 14:33:49.492168 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:49.492093 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-83.ec2.internal\" not found" Jul 15 14:33:49.512862 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.512828 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.519250 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.519229 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.ec2.internal" Jul 15 14:33:49.594335 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:49.594284 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-83.ec2.internal\" not found" Jul 15 14:33:49.676137 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.676111 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 14:33:49.695410 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:49.695377 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-83.ec2.internal\" not found" Jul 15 14:33:49.768347 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.768312 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 14:33:49.769007 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.768527 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 14:33:49.769007 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.768533 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 14:33:49.795821 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:49.795782 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-83.ec2.internal\" not found" Jul 15 14:33:49.850000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.849909 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 14:28:48 +0000 UTC" deadline="2028-03-13 20:41:23.402418038 +0000 UTC" Jul 15 14:33:49.850000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.849940 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14574h7m33.552480971s" Jul 15 14:33:49.896534 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:49.896499 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-83.ec2.internal\" not found" Jul 15 14:33:49.939970 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:49.939930 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 14:33:50.032867 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.032832 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal" Jul 15 14:33:50.052240 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.052212 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 14:33:50.054108 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.054090 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.ec2.internal" Jul 15 14:33:50.072593 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.072573 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 14:33:50.133597 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.133514 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 14:33:50.378416 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:50.378374 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ba0b7cefb501e9fe51d83336aec9d4.slice/crio-34784408b0e0667648e774e3471ce748058f72d1d60717c55766934afaa9b3ac WatchSource:0}: Error finding container 34784408b0e0667648e774e3471ce748058f72d1d60717c55766934afaa9b3ac: Status 404 returned error can't find the container with id 34784408b0e0667648e774e3471ce748058f72d1d60717c55766934afaa9b3ac Jul 15 14:33:50.382984 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.382968 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 14:33:50.534938 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:50.534912 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeee09bf5e958e41488ee1c3e17d2133.slice/crio-96c6f217300899b752907d6e721ee9e0dde876936fe96b598e678f664ab7aa3a WatchSource:0}: Error finding container 96c6f217300899b752907d6e721ee9e0dde876936fe96b598e678f664ab7aa3a: Status 404 returned error can't find the container with id 96c6f217300899b752907d6e721ee9e0dde876936fe96b598e678f664ab7aa3a Jul 15 14:33:50.812577 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.812550 2574 apiserver.go:52] "Watching apiserver" Jul 15 14:33:50.823491 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.823464 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 14:33:50.823970 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.823946 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-4hzj7","openshift-multus/multus-additional-cni-plugins-cd8gf","openshift-network-diagnostics/network-check-target-5rtjv","openshift-ovn-kubernetes/ovnkube-node-bdhfs","kube-system/global-pull-secret-syncer-rgz47","kube-system/kube-apiserver-proxy-ip-10-0-139-83.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f","openshift-cluster-node-tuning-operator/tuned-kbhnj","openshift-image-registry/node-ca-djz58","openshift-multus/network-metrics-daemon-9jbvd","openshift-network-operator/iptables-alerter-tnwkn","kube-system/konnectivity-agent-5cdr7","openshift-dns/node-resolver-6sxqv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal"] Jul 15 14:33:50.825586 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.825558 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.826731 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.826713 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6sxqv" Jul 15 14:33:50.827945 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.827919 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:33:50.828051 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:50.828021 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:33:50.828879 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.828862 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 14:33:50.828990 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.828865 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 14:33:50.829210 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.829182 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 14:33:50.829482 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.829411 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.829482 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.829434 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r8c9t\"" Jul 15 14:33:50.829626 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.829582 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 14:33:50.829679 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.829656 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 14:33:50.830592 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.830573 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-km7qp\"" Jul 15 14:33:50.830690 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.830681 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:50.830773 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:50.830750 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:33:50.832261 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.832239 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.832967 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.832943 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 14:33:50.833145 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.833122 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 14:33:50.833145 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.833152 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 14:33:50.833285 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.833163 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 14:33:50.834282 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.834263 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 14:33:50.834371 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.834279 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 14:33:50.834371 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.834288 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k6tnr\"" Jul 15 14:33:50.834371 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.834294 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 14:33:50.834985 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.834714 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-djz58" Jul 15 14:33:50.835617 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.835512 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 14:33:50.836012 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.835993 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.836635 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.836618 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 14:33:50.836715 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.836633 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7wsxn\"" Jul 15 14:33:50.837836 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.837728 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:50.840700 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.839561 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 14:33:50.840700 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.839618 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 14:33:50.840700 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.840367 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 14:33:50.840700 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.840384 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c2x9z\"" Jul 15 14:33:50.840700 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.840470 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dtmp6\"" Jul 15 14:33:50.841196 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.841176 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 14:33:50.841984 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.841295 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 14:33:50.841984 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.841382 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 14:33:50.841984 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.841192 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 14:33:50.841984 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.841194 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 14:33:50.841984 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.841266 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-thpdx\"" Jul 15 14:33:50.842374 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.842347 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 14:33:50.843734 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.843719 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:33:50.843832 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:50.843798 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:33:50.845156 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.845135 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tnwkn" Jul 15 14:33:50.847274 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847256 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5cdr7" Jul 15 14:33:50.847598 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847432 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-host\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.847598 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847466 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d017a037-beb6-4045-a169-75f928ba4cae-etc-tuned\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.847598 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847495 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-etc-openvswitch\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.847598 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-run-ovn-kubernetes\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.847598 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847546 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zld6z\" (UniqueName: \"kubernetes.io/projected/ee8a402a-6831-434d-b19d-d0a8617c5c91-kube-api-access-zld6z\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.847598 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847569 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-modprobe-d\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.847598 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847588 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-kubernetes\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.847994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-run\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.847994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847627 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-var-lib-kubelet\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.847994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847645 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g899l\" (UniqueName: \"kubernetes.io/projected/d017a037-beb6-4045-a169-75f928ba4cae-kube-api-access-g899l\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.847994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847682 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.847994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847721 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e85e4589-15f8-4790-b90f-246692b35897-kubelet-config\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:50.847994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847747 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-sysctl-d\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.847994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847769 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-registration-dir\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.847994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-device-dir\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.847994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847842 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-log-socket\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.847994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847899 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ee8a402a-6831-434d-b19d-d0a8617c5c91-ovnkube-config\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.847994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847931 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c59c38d-2211-4547-80df-7505bb4ef76d-host\") pod \"node-ca-djz58\" (UID: \"5c59c38d-2211-4547-80df-7505bb4ef76d\") " pod="openshift-image-registry/node-ca-djz58" Jul 15 14:33:50.847994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847966 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:50.847994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.847996 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-kubelet\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848017 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-run-systemd\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848030 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d017a037-beb6-4045-a169-75f928ba4cae-tmp\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848057 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-run-netns\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848096 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-run-openvswitch\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848139 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-run-ovn\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848163 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee8a402a-6831-434d-b19d-d0a8617c5c91-ovn-node-metrics-cert\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848189 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c59c38d-2211-4547-80df-7505bb4ef76d-serviceca\") pod \"node-ca-djz58\" (UID: \"5c59c38d-2211-4547-80df-7505bb4ef76d\") " pod="openshift-image-registry/node-ca-djz58" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848210 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50bab32c-4edf-4ebe-9bd0-cfa9ef0df269-hosts-file\") pod \"node-resolver-6sxqv\" (UID: \"50bab32c-4edf-4ebe-9bd0-cfa9ef0df269\") " pod="openshift-dns/node-resolver-6sxqv" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848231 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gvdv\" (UniqueName: \"kubernetes.io/projected/50bab32c-4edf-4ebe-9bd0-cfa9ef0df269-kube-api-access-5gvdv\") pod \"node-resolver-6sxqv\" (UID: \"50bab32c-4edf-4ebe-9bd0-cfa9ef0df269\") " pod="openshift-dns/node-resolver-6sxqv" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-socket-dir\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-systemd-units\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848321 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-slash\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848359 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-var-lib-openvswitch\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848382 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-cni-bin\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848405 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwntz\" (UniqueName: \"kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz\") pod \"network-check-target-5rtjv\" (UID: \"7371eff3-e8e5-467d-a2fd-77ebbb193507\") " pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848450 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-sys\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.848600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848480 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-lib-modules\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-node-log\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848525 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848656 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ee8a402a-6831-434d-b19d-d0a8617c5c91-ovnkube-script-lib\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848694 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbkn8\" (UniqueName: \"kubernetes.io/projected/5c59c38d-2211-4547-80df-7505bb4ef76d-kube-api-access-dbkn8\") pod \"node-ca-djz58\" (UID: \"5c59c38d-2211-4547-80df-7505bb4ef76d\") " pod="openshift-image-registry/node-ca-djz58" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848722 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848742 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50bab32c-4edf-4ebe-9bd0-cfa9ef0df269-tmp-dir\") pod \"node-resolver-6sxqv\" (UID: \"50bab32c-4edf-4ebe-9bd0-cfa9ef0df269\") " pod="openshift-dns/node-resolver-6sxqv" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848770 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e85e4589-15f8-4790-b90f-246692b35897-dbus\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848795 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-sysctl-conf\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848820 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-systemd\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848866 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-etc-selinux\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848932 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-sys-fs\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848962 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.848966 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9p6k\" (UniqueName: \"kubernetes.io/projected/e72acbe4-3923-4eae-8a56-0bb829dfebf1-kube-api-access-d9p6k\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.849006 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-cni-netd\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.849028 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ee8a402a-6831-434d-b19d-d0a8617c5c91-env-overrides\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.849056 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-sysconfig\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.849088 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.849135 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xqhdf\"" Jul 15 14:33:50.849325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.849162 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 14:33:50.850221 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.850139 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mwb7r\"" Jul 15 14:33:50.850221 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.850197 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 14:33:50.850676 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.850656 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 14:33:50.856376 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.856356 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 14:33:50.880945 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.880927 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b6ff8" Jul 15 14:33:50.890736 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.890717 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b6ff8" Jul 15 14:33:50.934796 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.934764 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 14:33:50.949258 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-lib-modules\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.949397 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949271 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ee8a402a-6831-434d-b19d-d0a8617c5c91-ovnkube-script-lib\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.949397 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2znk\" (UniqueName: \"kubernetes.io/projected/d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1-kube-api-access-p2znk\") pod \"iptables-alerter-tnwkn\" (UID: \"d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1\") " pod="openshift-network-operator/iptables-alerter-tnwkn" Jul 15 14:33:50.949397 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-multus-cni-dir\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.949397 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-run-multus-certs\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.949397 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949393 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e85e4589-15f8-4790-b90f-246692b35897-dbus\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:50.949642 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949416 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-lib-modules\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.949642 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-sysctl-conf\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.949642 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-etc-selinux\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.949642 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949492 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-sys-fs\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.949642 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949508 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9p6k\" (UniqueName: \"kubernetes.io/projected/e72acbe4-3923-4eae-8a56-0bb829dfebf1-kube-api-access-d9p6k\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.949642 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949514 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e85e4589-15f8-4790-b90f-246692b35897-dbus\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:50.949642 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949532 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-cni-netd\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.949642 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ee8a402a-6831-434d-b19d-d0a8617c5c91-env-overrides\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.949642 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949577 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-etc-selinux\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.949642 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949583 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zld6z\" (UniqueName: \"kubernetes.io/projected/ee8a402a-6831-434d-b19d-d0a8617c5c91-kube-api-access-zld6z\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.949642 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949584 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-sysctl-conf\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.949642 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949582 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-sys-fs\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.949642 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-host\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.949642 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949639 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-cni-netd\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949644 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d017a037-beb6-4045-a169-75f928ba4cae-etc-tuned\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-etc-openvswitch\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949702 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-host\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949709 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-run-ovn-kubernetes\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949738 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/960e795d-fb41-4f9d-8b1f-533f625dbb08-cni-binary-copy\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-os-release\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949776 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-etc-openvswitch\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949786 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-multus-socket-dir-parent\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-kubernetes\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949820 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-run-ovn-kubernetes\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949849 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-run\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949866 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-var-lib-kubelet\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949876 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-kubernetes\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949900 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g899l\" (UniqueName: \"kubernetes.io/projected/d017a037-beb6-4045-a169-75f928ba4cae-kube-api-access-g899l\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949924 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-run\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.950346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949941 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-var-lib-kubelet\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-cnibin\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949973 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ea1b539-6ec9-41b2-91aa-c27772f97451-multus-daemon-config\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.949999 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950002 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e85e4589-15f8-4790-b90f-246692b35897-kubelet-config\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950014 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-sysctl-d\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950046 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ee8a402a-6831-434d-b19d-d0a8617c5c91-ovnkube-script-lib\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950065 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-device-dir\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950017 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ee8a402a-6831-434d-b19d-d0a8617c5c91-env-overrides\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-log-socket\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ee8a402a-6831-434d-b19d-d0a8617c5c91-ovnkube-config\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950118 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e85e4589-15f8-4790-b90f-246692b35897-kubelet-config\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950153 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-device-dir\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950165 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c59c38d-2211-4547-80df-7505bb4ef76d-host\") pod \"node-ca-djz58\" (UID: \"5c59c38d-2211-4547-80df-7505bb4ef76d\") " pod="openshift-image-registry/node-ca-djz58" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950197 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-sysctl-d\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950205 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c59c38d-2211-4547-80df-7505bb4ef76d-host\") pod \"node-ca-djz58\" (UID: \"5c59c38d-2211-4547-80df-7505bb4ef76d\") " pod="openshift-image-registry/node-ca-djz58" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950241 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-log-socket\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.951051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950294 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-run-netns\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950339 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-var-lib-kubelet\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950367 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/960e795d-fb41-4f9d-8b1f-533f625dbb08-cnibin\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950392 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/960e795d-fb41-4f9d-8b1f-533f625dbb08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950409 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9j8f\" (UniqueName: \"kubernetes.io/projected/960e795d-fb41-4f9d-8b1f-533f625dbb08-kube-api-access-c9j8f\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950431 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-system-cni-dir\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950458 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-etc-kubernetes\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d017a037-beb6-4045-a169-75f928ba4cae-tmp\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-run-openvswitch\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950572 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c59c38d-2211-4547-80df-7505bb4ef76d-serviceca\") pod \"node-ca-djz58\" (UID: \"5c59c38d-2211-4547-80df-7505bb4ef76d\") " pod="openshift-image-registry/node-ca-djz58" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950598 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/960e795d-fb41-4f9d-8b1f-533f625dbb08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950650 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ee8a402a-6831-434d-b19d-d0a8617c5c91-ovnkube-config\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-run-k8s-cni-cncf-io\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-systemd-units\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950772 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-slash\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-var-lib-openvswitch\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950863 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-sys\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.951847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950904 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-node-log\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950928 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.950963 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-slash\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951003 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-var-lib-openvswitch\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbkn8\" (UniqueName: \"kubernetes.io/projected/5c59c38d-2211-4547-80df-7505bb4ef76d-kube-api-access-dbkn8\") pod \"node-ca-djz58\" (UID: \"5c59c38d-2211-4547-80df-7505bb4ef76d\") " pod="openshift-image-registry/node-ca-djz58" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951037 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-systemd-units\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951030 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951072 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5c59c38d-2211-4547-80df-7505bb4ef76d-serviceca\") pod \"node-ca-djz58\" (UID: \"5c59c38d-2211-4547-80df-7505bb4ef76d\") " pod="openshift-image-registry/node-ca-djz58" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951079 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-run-openvswitch\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951084 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-sys\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951076 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-node-log\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951109 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50bab32c-4edf-4ebe-9bd0-cfa9ef0df269-tmp-dir\") pod \"node-resolver-6sxqv\" (UID: \"50bab32c-4edf-4ebe-9bd0-cfa9ef0df269\") " pod="openshift-dns/node-resolver-6sxqv" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951136 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ea1b539-6ec9-41b2-91aa-c27772f97451-cni-binary-copy\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951154 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-hostroot\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-systemd\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1-host-slash\") pod \"iptables-alerter-tnwkn\" (UID: \"d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1\") " pod="openshift-network-operator/iptables-alerter-tnwkn" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951215 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac545d6a-d123-455e-80d4-890435b906a7-agent-certs\") pod \"konnectivity-agent-5cdr7\" (UID: \"ac545d6a-d123-455e-80d4-890435b906a7\") " pod="kube-system/konnectivity-agent-5cdr7" Jul 15 14:33:50.952582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951235 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-systemd\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951249 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n76vl\" (UniqueName: \"kubernetes.io/projected/0ea1b539-6ec9-41b2-91aa-c27772f97451-kube-api-access-n76vl\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951282 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-sysconfig\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951310 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/960e795d-fb41-4f9d-8b1f-533f625dbb08-system-cni-dir\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951344 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1-iptables-alerter-script\") pod \"iptables-alerter-tnwkn\" (UID: \"d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1\") " pod="openshift-network-operator/iptables-alerter-tnwkn" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951370 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50bab32c-4edf-4ebe-9bd0-cfa9ef0df269-tmp-dir\") pod \"node-resolver-6sxqv\" (UID: \"50bab32c-4edf-4ebe-9bd0-cfa9ef0df269\") " pod="openshift-dns/node-resolver-6sxqv" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951398 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-sysconfig\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951372 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-var-lib-cni-multus\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-modprobe-d\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951462 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-multus-conf-dir\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951486 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-registration-dir\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951512 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/960e795d-fb41-4f9d-8b1f-533f625dbb08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951532 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d017a037-beb6-4045-a169-75f928ba4cae-etc-modprobe-d\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951539 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs\") pod \"network-metrics-daemon-9jbvd\" (UID: \"7d23262f-6be8-46f4-9122-28ce9683ed4a\") " pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951571 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951584 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-registration-dir\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-kubelet\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.953220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951622 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-run-systemd\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951656 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac545d6a-d123-455e-80d4-890435b906a7-konnectivity-ca\") pod \"konnectivity-agent-5cdr7\" (UID: \"ac545d6a-d123-455e-80d4-890435b906a7\") " pod="kube-system/konnectivity-agent-5cdr7" Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-run-netns\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:50.951693 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951711 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-run-ovn\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951735 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee8a402a-6831-434d-b19d-d0a8617c5c91-ovn-node-metrics-cert\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:50.951773 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret podName:e85e4589-15f8-4790-b90f-246692b35897 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:51.451734521 +0000 UTC m=+4.157810608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret") pod "global-pull-secret-syncer-rgz47" (UID: "e85e4589-15f8-4790-b90f-246692b35897") : object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951778 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-run-systemd\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50bab32c-4edf-4ebe-9bd0-cfa9ef0df269-hosts-file\") pod \"node-resolver-6sxqv\" (UID: \"50bab32c-4edf-4ebe-9bd0-cfa9ef0df269\") " pod="openshift-dns/node-resolver-6sxqv" Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951855 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gvdv\" (UniqueName: \"kubernetes.io/projected/50bab32c-4edf-4ebe-9bd0-cfa9ef0df269-kube-api-access-5gvdv\") pod \"node-resolver-6sxqv\" (UID: \"50bab32c-4edf-4ebe-9bd0-cfa9ef0df269\") " pod="openshift-dns/node-resolver-6sxqv" Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951898 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbdmz\" (UniqueName: \"kubernetes.io/projected/7d23262f-6be8-46f4-9122-28ce9683ed4a-kube-api-access-sbdmz\") pod \"network-metrics-daemon-9jbvd\" (UID: \"7d23262f-6be8-46f4-9122-28ce9683ed4a\") " pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951922 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-socket-dir\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-cni-bin\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951965 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-run-netns\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.951972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwntz\" (UniqueName: \"kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz\") pod \"network-check-target-5rtjv\" (UID: \"7371eff3-e8e5-467d-a2fd-77ebbb193507\") " pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.952006 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-run-ovn\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.952007 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/960e795d-fb41-4f9d-8b1f-533f625dbb08-os-release\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:50.954000 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.952187 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-kubelet\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.954636 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.952229 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee8a402a-6831-434d-b19d-d0a8617c5c91-host-cni-bin\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.954636 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.952262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-var-lib-cni-bin\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:50.954636 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.952275 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e72acbe4-3923-4eae-8a56-0bb829dfebf1-socket-dir\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.954636 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.952296 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50bab32c-4edf-4ebe-9bd0-cfa9ef0df269-hosts-file\") pod \"node-resolver-6sxqv\" (UID: \"50bab32c-4edf-4ebe-9bd0-cfa9ef0df269\") " pod="openshift-dns/node-resolver-6sxqv" Jul 15 14:33:50.954636 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.953731 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d017a037-beb6-4045-a169-75f928ba4cae-etc-tuned\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.954636 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.953804 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d017a037-beb6-4045-a169-75f928ba4cae-tmp\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.954636 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.954201 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ee8a402a-6831-434d-b19d-d0a8617c5c91-ovn-node-metrics-cert\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.969816 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.969794 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g899l\" (UniqueName: \"kubernetes.io/projected/d017a037-beb6-4045-a169-75f928ba4cae-kube-api-access-g899l\") pod \"tuned-kbhnj\" (UID: \"d017a037-beb6-4045-a169-75f928ba4cae\") " pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:50.970105 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.970089 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9p6k\" (UniqueName: \"kubernetes.io/projected/e72acbe4-3923-4eae-8a56-0bb829dfebf1-kube-api-access-d9p6k\") pod \"aws-ebs-csi-driver-node-2r88f\" (UID: \"e72acbe4-3923-4eae-8a56-0bb829dfebf1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:50.971780 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:50.971757 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 14:33:50.971906 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:50.971786 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 14:33:50.971906 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:50.971799 2574 projected.go:196] Error preparing data for projected volume kube-api-access-vwntz for pod openshift-network-diagnostics/network-check-target-5rtjv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:50.972029 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:50.971914 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz podName:7371eff3-e8e5-467d-a2fd-77ebbb193507 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:51.471880964 +0000 UTC m=+4.177957050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vwntz" (UniqueName: "kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz") pod "network-check-target-5rtjv" (UID: "7371eff3-e8e5-467d-a2fd-77ebbb193507") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:50.972981 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.972962 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbkn8\" (UniqueName: \"kubernetes.io/projected/5c59c38d-2211-4547-80df-7505bb4ef76d-kube-api-access-dbkn8\") pod \"node-ca-djz58\" (UID: \"5c59c38d-2211-4547-80df-7505bb4ef76d\") " pod="openshift-image-registry/node-ca-djz58" Jul 15 14:33:50.973085 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.973010 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zld6z\" (UniqueName: \"kubernetes.io/projected/ee8a402a-6831-434d-b19d-d0a8617c5c91-kube-api-access-zld6z\") pod \"ovnkube-node-bdhfs\" (UID: \"ee8a402a-6831-434d-b19d-d0a8617c5c91\") " pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:50.973085 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:50.973018 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gvdv\" (UniqueName: \"kubernetes.io/projected/50bab32c-4edf-4ebe-9bd0-cfa9ef0df269-kube-api-access-5gvdv\") pod \"node-resolver-6sxqv\" (UID: \"50bab32c-4edf-4ebe-9bd0-cfa9ef0df269\") " pod="openshift-dns/node-resolver-6sxqv" Jul 15 14:33:51.052631 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052599 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/960e795d-fb41-4f9d-8b1f-533f625dbb08-cni-binary-copy\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.052800 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052639 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-os-release\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.052800 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052665 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-multus-socket-dir-parent\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.052800 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052693 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-cnibin\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.052800 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ea1b539-6ec9-41b2-91aa-c27772f97451-multus-daemon-config\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.052800 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052743 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-run-netns\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.052800 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052765 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-var-lib-kubelet\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.052800 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052801 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/960e795d-fb41-4f9d-8b1f-533f625dbb08-cnibin\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.053153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052826 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/960e795d-fb41-4f9d-8b1f-533f625dbb08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.053153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052852 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9j8f\" (UniqueName: \"kubernetes.io/projected/960e795d-fb41-4f9d-8b1f-533f625dbb08-kube-api-access-c9j8f\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.053153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052877 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-system-cni-dir\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.053153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052914 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-etc-kubernetes\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.053153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052942 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/960e795d-fb41-4f9d-8b1f-533f625dbb08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.053153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052962 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-run-k8s-cni-cncf-io\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.053153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052982 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ea1b539-6ec9-41b2-91aa-c27772f97451-cni-binary-copy\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.053153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.052996 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-hostroot\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.053153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053012 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1-host-slash\") pod \"iptables-alerter-tnwkn\" (UID: \"d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1\") " pod="openshift-network-operator/iptables-alerter-tnwkn" Jul 15 14:33:51.053153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac545d6a-d123-455e-80d4-890435b906a7-agent-certs\") pod \"konnectivity-agent-5cdr7\" (UID: \"ac545d6a-d123-455e-80d4-890435b906a7\") " pod="kube-system/konnectivity-agent-5cdr7" Jul 15 14:33:51.053153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n76vl\" (UniqueName: \"kubernetes.io/projected/0ea1b539-6ec9-41b2-91aa-c27772f97451-kube-api-access-n76vl\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.053153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/960e795d-fb41-4f9d-8b1f-533f625dbb08-system-cni-dir\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.053153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1-iptables-alerter-script\") pod \"iptables-alerter-tnwkn\" (UID: \"d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1\") " pod="openshift-network-operator/iptables-alerter-tnwkn" Jul 15 14:33:51.053153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-var-lib-cni-multus\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.053153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053157 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-multus-conf-dir\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.053816 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053211 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-multus-conf-dir\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.053816 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053212 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-system-cni-dir\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.053816 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053220 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-hostroot\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.053816 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053223 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1-host-slash\") pod \"iptables-alerter-tnwkn\" (UID: \"d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1\") " pod="openshift-network-operator/iptables-alerter-tnwkn" Jul 15 14:33:51.053816 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053275 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/960e795d-fb41-4f9d-8b1f-533f625dbb08-system-cni-dir\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.053816 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053306 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/960e795d-fb41-4f9d-8b1f-533f625dbb08-cnibin\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.053816 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053329 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-multus-socket-dir-parent\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.053816 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053380 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-os-release\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.053816 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053414 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-etc-kubernetes\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.053816 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053496 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/960e795d-fb41-4f9d-8b1f-533f625dbb08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.053816 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053556 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-cnibin\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.053816 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053591 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-var-lib-cni-multus\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.053999 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ea1b539-6ec9-41b2-91aa-c27772f97451-multus-daemon-config\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054003 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ea1b539-6ec9-41b2-91aa-c27772f97451-cni-binary-copy\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054002 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/960e795d-fb41-4f9d-8b1f-533f625dbb08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054032 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/960e795d-fb41-4f9d-8b1f-533f625dbb08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs\") pod \"network-metrics-daemon-9jbvd\" (UID: \"7d23262f-6be8-46f4-9122-28ce9683ed4a\") " pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054076 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-var-lib-kubelet\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054075 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-run-k8s-cni-cncf-io\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054097 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1-iptables-alerter-script\") pod \"iptables-alerter-tnwkn\" (UID: \"d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1\") " pod="openshift-network-operator/iptables-alerter-tnwkn" Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054121 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac545d6a-d123-455e-80d4-890435b906a7-konnectivity-ca\") pod \"konnectivity-agent-5cdr7\" (UID: \"ac545d6a-d123-455e-80d4-890435b906a7\") " pod="kube-system/konnectivity-agent-5cdr7" Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:51.054160 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054163 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbdmz\" (UniqueName: \"kubernetes.io/projected/7d23262f-6be8-46f4-9122-28ce9683ed4a-kube-api-access-sbdmz\") pod \"network-metrics-daemon-9jbvd\" (UID: \"7d23262f-6be8-46f4-9122-28ce9683ed4a\") " pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054204 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/960e795d-fb41-4f9d-8b1f-533f625dbb08-os-release\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:51.054224 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs podName:7d23262f-6be8-46f4-9122-28ce9683ed4a nodeName:}" failed. No retries permitted until 2026-07-15 14:33:51.554206159 +0000 UTC m=+4.260282260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs") pod "network-metrics-daemon-9jbvd" (UID: "7d23262f-6be8-46f4-9122-28ce9683ed4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054254 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-var-lib-cni-bin\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054263 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/960e795d-fb41-4f9d-8b1f-533f625dbb08-os-release\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054269 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-run-netns\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.054381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054285 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2znk\" (UniqueName: \"kubernetes.io/projected/d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1-kube-api-access-p2znk\") pod \"iptables-alerter-tnwkn\" (UID: \"d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1\") " pod="openshift-network-operator/iptables-alerter-tnwkn" Jul 15 14:33:51.055102 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054312 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-multus-cni-dir\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.055102 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054334 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-run-multus-certs\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.055102 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054162 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/960e795d-fb41-4f9d-8b1f-533f625dbb08-cni-binary-copy\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.055102 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054411 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-run-multus-certs\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.055102 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054446 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-multus-cni-dir\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.055102 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054507 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/960e795d-fb41-4f9d-8b1f-533f625dbb08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.055102 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054566 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ea1b539-6ec9-41b2-91aa-c27772f97451-host-var-lib-cni-bin\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.055102 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.054601 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac545d6a-d123-455e-80d4-890435b906a7-konnectivity-ca\") pod \"konnectivity-agent-5cdr7\" (UID: \"ac545d6a-d123-455e-80d4-890435b906a7\") " pod="kube-system/konnectivity-agent-5cdr7" Jul 15 14:33:51.056080 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.056061 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac545d6a-d123-455e-80d4-890435b906a7-agent-certs\") pod \"konnectivity-agent-5cdr7\" (UID: \"ac545d6a-d123-455e-80d4-890435b906a7\") " pod="kube-system/konnectivity-agent-5cdr7" Jul 15 14:33:51.066686 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.066628 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n76vl\" (UniqueName: \"kubernetes.io/projected/0ea1b539-6ec9-41b2-91aa-c27772f97451-kube-api-access-n76vl\") pod \"multus-4hzj7\" (UID: \"0ea1b539-6ec9-41b2-91aa-c27772f97451\") " pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.067704 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.067676 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9j8f\" (UniqueName: \"kubernetes.io/projected/960e795d-fb41-4f9d-8b1f-533f625dbb08-kube-api-access-c9j8f\") pod \"multus-additional-cni-plugins-cd8gf\" (UID: \"960e795d-fb41-4f9d-8b1f-533f625dbb08\") " pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.068081 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.068062 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbdmz\" (UniqueName: \"kubernetes.io/projected/7d23262f-6be8-46f4-9122-28ce9683ed4a-kube-api-access-sbdmz\") pod \"network-metrics-daemon-9jbvd\" (UID: \"7d23262f-6be8-46f4-9122-28ce9683ed4a\") " pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:33:51.069055 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.069026 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2znk\" (UniqueName: \"kubernetes.io/projected/d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1-kube-api-access-p2znk\") pod \"iptables-alerter-tnwkn\" (UID: \"d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1\") " pod="openshift-network-operator/iptables-alerter-tnwkn" Jul 15 14:33:51.081679 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.081636 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.ec2.internal" event={"ID":"eeee09bf5e958e41488ee1c3e17d2133","Type":"ContainerStarted","Data":"96c6f217300899b752907d6e721ee9e0dde876936fe96b598e678f664ab7aa3a"} Jul 15 14:33:51.082832 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.082799 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal" event={"ID":"a8ba0b7cefb501e9fe51d83336aec9d4","Type":"ContainerStarted","Data":"34784408b0e0667648e774e3471ce748058f72d1d60717c55766934afaa9b3ac"} Jul 15 14:33:51.137589 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.137562 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" Jul 15 14:33:51.145986 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.145965 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6sxqv" Jul 15 14:33:51.147745 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:51.147717 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72acbe4_3923_4eae_8a56_0bb829dfebf1.slice/crio-e570f7654c74e34821bf913f27c27a7628e3f29a2394244c9e1e285fa001e788 WatchSource:0}: Error finding container e570f7654c74e34821bf913f27c27a7628e3f29a2394244c9e1e285fa001e788: Status 404 returned error can't find the container with id e570f7654c74e34821bf913f27c27a7628e3f29a2394244c9e1e285fa001e788 Jul 15 14:33:51.154761 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:51.154733 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50bab32c_4edf_4ebe_9bd0_cfa9ef0df269.slice/crio-273664643c309bfebf72cd488844c8d859a7242771352d716d29df09221a8fdc WatchSource:0}: Error finding container 273664643c309bfebf72cd488844c8d859a7242771352d716d29df09221a8fdc: Status 404 returned error can't find the container with id 273664643c309bfebf72cd488844c8d859a7242771352d716d29df09221a8fdc Jul 15 14:33:51.155958 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.155930 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:33:51.160425 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.160399 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" Jul 15 14:33:51.165154 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:51.165127 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8a402a_6831_434d_b19d_d0a8617c5c91.slice/crio-9dc2684722f666982750b8ae55aac3db25138b1f9d695e27d72c078c214a5d73 WatchSource:0}: Error finding container 9dc2684722f666982750b8ae55aac3db25138b1f9d695e27d72c078c214a5d73: Status 404 returned error can't find the container with id 9dc2684722f666982750b8ae55aac3db25138b1f9d695e27d72c078c214a5d73 Jul 15 14:33:51.167295 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.167278 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-djz58" Jul 15 14:33:51.171488 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:51.171466 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd017a037_beb6_4045_a169_75f928ba4cae.slice/crio-199433e4bc7c5685bc365f95205aa566597a344e0c73530b671eaf24e648ab8b WatchSource:0}: Error finding container 199433e4bc7c5685bc365f95205aa566597a344e0c73530b671eaf24e648ab8b: Status 404 returned error can't find the container with id 199433e4bc7c5685bc365f95205aa566597a344e0c73530b671eaf24e648ab8b Jul 15 14:33:51.173358 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.173325 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4hzj7" Jul 15 14:33:51.179582 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:51.179550 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c59c38d_2211_4547_80df_7505bb4ef76d.slice/crio-6852bcad930e32be8e9c1b1a46c9cf230014caa7c8941615acf26bbe765b5730 WatchSource:0}: Error finding container 6852bcad930e32be8e9c1b1a46c9cf230014caa7c8941615acf26bbe765b5730: Status 404 returned error can't find the container with id 6852bcad930e32be8e9c1b1a46c9cf230014caa7c8941615acf26bbe765b5730 Jul 15 14:33:51.181017 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.180948 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cd8gf" Jul 15 14:33:51.187986 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.187766 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tnwkn" Jul 15 14:33:51.190650 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:51.190623 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960e795d_fb41_4f9d_8b1f_533f625dbb08.slice/crio-5693a6c48b11680f04df2026b6fd0a29cb4b8e147f45318bb1cff2a4b1abaed8 WatchSource:0}: Error finding container 5693a6c48b11680f04df2026b6fd0a29cb4b8e147f45318bb1cff2a4b1abaed8: Status 404 returned error can't find the container with id 5693a6c48b11680f04df2026b6fd0a29cb4b8e147f45318bb1cff2a4b1abaed8 Jul 15 14:33:51.193172 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.193157 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5cdr7" Jul 15 14:33:51.196965 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:51.196934 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7bcb32a_c6a9_4ca7_9cf7_98ad6ec1fea1.slice/crio-79b6d069eb9cf645a70295dd498f3ef436effd286c39a240fdf99ab33e857e9d WatchSource:0}: Error finding container 79b6d069eb9cf645a70295dd498f3ef436effd286c39a240fdf99ab33e857e9d: Status 404 returned error can't find the container with id 79b6d069eb9cf645a70295dd498f3ef436effd286c39a240fdf99ab33e857e9d Jul 15 14:33:51.457364 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.457276 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:51.457524 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:51.457461 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:51.457524 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:51.457525 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret podName:e85e4589-15f8-4790-b90f-246692b35897 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:52.457506993 +0000 UTC m=+5.163583090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret") pod "global-pull-secret-syncer-rgz47" (UID: "e85e4589-15f8-4790-b90f-246692b35897") : object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:51.558193 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.558153 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs\") pod \"network-metrics-daemon-9jbvd\" (UID: \"7d23262f-6be8-46f4-9122-28ce9683ed4a\") " pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:33:51.558368 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:51.558321 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:51.558427 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.558414 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwntz\" (UniqueName: \"kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz\") pod \"network-check-target-5rtjv\" (UID: \"7371eff3-e8e5-467d-a2fd-77ebbb193507\") " pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:33:51.558490 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:51.558476 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs podName:7d23262f-6be8-46f4-9122-28ce9683ed4a nodeName:}" failed. No retries permitted until 2026-07-15 14:33:52.558451428 +0000 UTC m=+5.264527515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs") pod "network-metrics-daemon-9jbvd" (UID: "7d23262f-6be8-46f4-9122-28ce9683ed4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:51.558579 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:51.558566 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 14:33:51.558638 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:51.558587 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 14:33:51.558638 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:51.558601 2574 projected.go:196] Error preparing data for projected volume kube-api-access-vwntz for pod openshift-network-diagnostics/network-check-target-5rtjv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:51.558734 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:51.558652 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz podName:7371eff3-e8e5-467d-a2fd-77ebbb193507 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:52.558638381 +0000 UTC m=+5.264714468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwntz" (UniqueName: "kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz") pod "network-check-target-5rtjv" (UID: "7371eff3-e8e5-467d-a2fd-77ebbb193507") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:51.599805 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:33:51.599770 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac545d6a_d123_455e_80d4_890435b906a7.slice/crio-cb49c9b19cf9b92245cf92e83ee59915833707529248f82df1e58ed2ea7a444d WatchSource:0}: Error finding container cb49c9b19cf9b92245cf92e83ee59915833707529248f82df1e58ed2ea7a444d: Status 404 returned error can't find the container with id cb49c9b19cf9b92245cf92e83ee59915833707529248f82df1e58ed2ea7a444d Jul 15 14:33:51.892947 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.892864 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 14:28:50 +0000 UTC" deadline="2027-12-25 09:32:22.896352747 +0000 UTC" Jul 15 14:33:51.892947 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:51.892946 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12666h58m31.003411842s" Jul 15 14:33:52.078092 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.077851 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:33:52.078274 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:52.078199 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:33:52.087821 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.087750 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tnwkn" event={"ID":"d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1","Type":"ContainerStarted","Data":"79b6d069eb9cf645a70295dd498f3ef436effd286c39a240fdf99ab33e857e9d"} Jul 15 14:33:52.089254 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.089206 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4hzj7" event={"ID":"0ea1b539-6ec9-41b2-91aa-c27772f97451","Type":"ContainerStarted","Data":"a2722c42ec044ca723093796e0f0770684b9694a37f27c2cbfdbb9fb47d45381"} Jul 15 14:33:52.090569 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.090504 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" event={"ID":"d017a037-beb6-4045-a169-75f928ba4cae","Type":"ContainerStarted","Data":"199433e4bc7c5685bc365f95205aa566597a344e0c73530b671eaf24e648ab8b"} Jul 15 14:33:52.092081 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.092055 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" event={"ID":"e72acbe4-3923-4eae-8a56-0bb829dfebf1","Type":"ContainerStarted","Data":"e570f7654c74e34821bf913f27c27a7628e3f29a2394244c9e1e285fa001e788"} Jul 15 14:33:52.093807 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.093780 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5cdr7" event={"ID":"ac545d6a-d123-455e-80d4-890435b906a7","Type":"ContainerStarted","Data":"cb49c9b19cf9b92245cf92e83ee59915833707529248f82df1e58ed2ea7a444d"} Jul 15 14:33:52.095214 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.095190 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cd8gf" event={"ID":"960e795d-fb41-4f9d-8b1f-533f625dbb08","Type":"ContainerStarted","Data":"5693a6c48b11680f04df2026b6fd0a29cb4b8e147f45318bb1cff2a4b1abaed8"} Jul 15 14:33:52.096523 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.096481 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-djz58" event={"ID":"5c59c38d-2211-4547-80df-7505bb4ef76d","Type":"ContainerStarted","Data":"6852bcad930e32be8e9c1b1a46c9cf230014caa7c8941615acf26bbe765b5730"} Jul 15 14:33:52.097920 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.097824 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" event={"ID":"ee8a402a-6831-434d-b19d-d0a8617c5c91","Type":"ContainerStarted","Data":"9dc2684722f666982750b8ae55aac3db25138b1f9d695e27d72c078c214a5d73"} Jul 15 14:33:52.099033 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.099011 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6sxqv" event={"ID":"50bab32c-4edf-4ebe-9bd0-cfa9ef0df269","Type":"ContainerStarted","Data":"273664643c309bfebf72cd488844c8d859a7242771352d716d29df09221a8fdc"} Jul 15 14:33:52.465954 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.465843 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:52.466039 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:52.466028 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:52.466095 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:52.466087 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret podName:e85e4589-15f8-4790-b90f-246692b35897 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:54.466069886 +0000 UTC m=+7.172145975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret") pod "global-pull-secret-syncer-rgz47" (UID: "e85e4589-15f8-4790-b90f-246692b35897") : object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:52.567233 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.567194 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs\") pod \"network-metrics-daemon-9jbvd\" (UID: \"7d23262f-6be8-46f4-9122-28ce9683ed4a\") " pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:33:52.567364 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.567267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwntz\" (UniqueName: \"kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz\") pod \"network-check-target-5rtjv\" (UID: \"7371eff3-e8e5-467d-a2fd-77ebbb193507\") " pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:33:52.567505 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:52.567487 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 14:33:52.567579 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:52.567514 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 14:33:52.567579 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:52.567527 2574 projected.go:196] Error preparing data for projected volume kube-api-access-vwntz for pod openshift-network-diagnostics/network-check-target-5rtjv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:52.567673 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:52.567582 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz podName:7371eff3-e8e5-467d-a2fd-77ebbb193507 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:54.567564146 +0000 UTC m=+7.273640231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwntz" (UniqueName: "kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz") pod "network-check-target-5rtjv" (UID: "7371eff3-e8e5-467d-a2fd-77ebbb193507") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:52.568125 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:52.568108 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:52.568234 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:52.568185 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs podName:7d23262f-6be8-46f4-9122-28ce9683ed4a nodeName:}" failed. No retries permitted until 2026-07-15 14:33:54.568157271 +0000 UTC m=+7.274233354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs") pod "network-metrics-daemon-9jbvd" (UID: "7d23262f-6be8-46f4-9122-28ce9683ed4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:52.887568 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.887534 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 14:33:52.893086 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.893052 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 14:28:50 +0000 UTC" deadline="2028-01-15 16:33:45.900983911 +0000 UTC" Jul 15 14:33:52.893086 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:52.893087 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13177h59m53.007901996s" Jul 15 14:33:53.077995 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:53.077968 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:53.078289 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:53.077987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:33:53.078741 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:53.078688 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:33:53.079291 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:53.079269 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:33:53.116930 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:53.116596 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.ec2.internal" event={"ID":"eeee09bf5e958e41488ee1c3e17d2133","Type":"ContainerStarted","Data":"bf8e654c729c2f539fb2a6d4abdd0500491e85d80c231e8485fc5062413a6fb0"} Jul 15 14:33:54.077789 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:54.077727 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:33:54.078258 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:54.077868 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:33:54.125059 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:54.125014 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal" event={"ID":"a8ba0b7cefb501e9fe51d83336aec9d4","Type":"ContainerStarted","Data":"8403720076e03cf67747bc3faec1196670cc9aae4d13ebdd2d342680b6c02982"} Jul 15 14:33:54.144535 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:54.144479 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-83.ec2.internal" podStartSLOduration=4.144456094 podStartE2EDuration="4.144456094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:33:53.136634412 +0000 UTC m=+5.842710518" watchObservedRunningTime="2026-07-15 14:33:54.144456094 +0000 UTC m=+6.850532200" Jul 15 14:33:54.486742 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:54.486103 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:54.486742 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:54.486287 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:54.486742 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:54.486356 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret podName:e85e4589-15f8-4790-b90f-246692b35897 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:58.486336351 +0000 UTC m=+11.192412440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret") pod "global-pull-secret-syncer-rgz47" (UID: "e85e4589-15f8-4790-b90f-246692b35897") : object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:54.587466 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:54.587429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs\") pod \"network-metrics-daemon-9jbvd\" (UID: \"7d23262f-6be8-46f4-9122-28ce9683ed4a\") " pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:33:54.587655 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:54.587497 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwntz\" (UniqueName: \"kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz\") pod \"network-check-target-5rtjv\" (UID: \"7371eff3-e8e5-467d-a2fd-77ebbb193507\") " pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:33:54.587718 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:54.587654 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 14:33:54.587718 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:54.587672 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 14:33:54.587718 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:54.587685 2574 projected.go:196] Error preparing data for projected volume kube-api-access-vwntz for pod openshift-network-diagnostics/network-check-target-5rtjv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:54.587867 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:54.587744 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz podName:7371eff3-e8e5-467d-a2fd-77ebbb193507 nodeName:}" failed. No retries permitted until 2026-07-15 14:33:58.587725782 +0000 UTC m=+11.293801867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwntz" (UniqueName: "kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz") pod "network-check-target-5rtjv" (UID: "7371eff3-e8e5-467d-a2fd-77ebbb193507") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:54.588182 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:54.588147 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:54.588286 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:54.588213 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs podName:7d23262f-6be8-46f4-9122-28ce9683ed4a nodeName:}" failed. No retries permitted until 2026-07-15 14:33:58.58819618 +0000 UTC m=+11.294272270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs") pod "network-metrics-daemon-9jbvd" (UID: "7d23262f-6be8-46f4-9122-28ce9683ed4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:55.079330 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:55.078792 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:55.079330 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:55.078949 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:33:55.080119 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:55.079946 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:33:55.080119 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:55.080066 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:33:56.078023 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:56.077988 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:33:56.078210 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:56.078117 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:33:56.132578 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:56.131902 2574 generic.go:356] "Generic (PLEG): container finished" podID="a8ba0b7cefb501e9fe51d83336aec9d4" containerID="8403720076e03cf67747bc3faec1196670cc9aae4d13ebdd2d342680b6c02982" exitCode=0 Jul 15 14:33:56.132578 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:56.131964 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal" event={"ID":"a8ba0b7cefb501e9fe51d83336aec9d4","Type":"ContainerDied","Data":"8403720076e03cf67747bc3faec1196670cc9aae4d13ebdd2d342680b6c02982"} Jul 15 14:33:57.077830 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:57.077492 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:57.077830 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:57.077497 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:33:57.077830 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:57.077628 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:33:57.077830 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:57.077693 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:33:58.078091 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:58.078045 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:33:58.078510 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:58.078186 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:33:58.520527 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:58.520485 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:58.520708 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:58.520677 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:58.520774 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:58.520753 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret podName:e85e4589-15f8-4790-b90f-246692b35897 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:06.520731879 +0000 UTC m=+19.226807978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret") pod "global-pull-secret-syncer-rgz47" (UID: "e85e4589-15f8-4790-b90f-246692b35897") : object "kube-system"/"original-pull-secret" not registered Jul 15 14:33:58.621253 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:58.621210 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs\") pod \"network-metrics-daemon-9jbvd\" (UID: \"7d23262f-6be8-46f4-9122-28ce9683ed4a\") " pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:33:58.621427 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:58.621272 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwntz\" (UniqueName: \"kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz\") pod \"network-check-target-5rtjv\" (UID: \"7371eff3-e8e5-467d-a2fd-77ebbb193507\") " pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:33:58.621427 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:58.621370 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:58.621427 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:58.621413 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 14:33:58.621615 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:58.621432 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 14:33:58.621615 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:58.621445 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs podName:7d23262f-6be8-46f4-9122-28ce9683ed4a nodeName:}" failed. No retries permitted until 2026-07-15 14:34:06.621424731 +0000 UTC m=+19.327500817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs") pod "network-metrics-daemon-9jbvd" (UID: "7d23262f-6be8-46f4-9122-28ce9683ed4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:33:58.621615 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:58.621446 2574 projected.go:196] Error preparing data for projected volume kube-api-access-vwntz for pod openshift-network-diagnostics/network-check-target-5rtjv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:58.621615 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:58.621488 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz podName:7371eff3-e8e5-467d-a2fd-77ebbb193507 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:06.621477709 +0000 UTC m=+19.327553792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwntz" (UniqueName: "kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz") pod "network-check-target-5rtjv" (UID: "7371eff3-e8e5-467d-a2fd-77ebbb193507") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:33:59.079058 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:59.078983 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:33:59.079449 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:59.079108 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:33:59.079493 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:33:59.079464 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:33:59.079576 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:33:59.079552 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:34:00.077971 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:00.077935 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:00.078162 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:00.078070 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:34:01.077442 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:01.077407 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:01.077851 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:01.077448 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:01.077851 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:01.077526 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:34:01.077851 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:01.077660 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:34:02.077486 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:02.077459 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:02.077796 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:02.077552 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:34:03.078267 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:03.078236 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:03.078713 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:03.078236 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:03.078713 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:03.078383 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:34:03.078713 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:03.078465 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:34:04.077881 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:04.077849 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:04.078046 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:04.077968 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:34:05.078091 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:05.078057 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:05.078476 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:05.078057 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:05.078476 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:05.078194 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:34:05.078476 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:05.078256 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:34:06.077250 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:06.077215 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:06.077439 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:06.077326 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:34:06.584611 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:06.584575 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:06.585107 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:06.584716 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 14:34:06.585107 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:06.584775 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret podName:e85e4589-15f8-4790-b90f-246692b35897 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:22.584758921 +0000 UTC m=+35.290835004 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret") pod "global-pull-secret-syncer-rgz47" (UID: "e85e4589-15f8-4790-b90f-246692b35897") : object "kube-system"/"original-pull-secret" not registered Jul 15 14:34:06.685593 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:06.685558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwntz\" (UniqueName: \"kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz\") pod \"network-check-target-5rtjv\" (UID: \"7371eff3-e8e5-467d-a2fd-77ebbb193507\") " pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:06.685765 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:06.685624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs\") pod \"network-metrics-daemon-9jbvd\" (UID: \"7d23262f-6be8-46f4-9122-28ce9683ed4a\") " pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:06.685765 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:06.685735 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:34:06.685765 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:06.685748 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 14:34:06.685912 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:06.685771 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 14:34:06.685912 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:06.685783 2574 projected.go:196] Error preparing data for projected volume kube-api-access-vwntz for pod openshift-network-diagnostics/network-check-target-5rtjv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:34:06.685912 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:06.685793 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs podName:7d23262f-6be8-46f4-9122-28ce9683ed4a nodeName:}" failed. No retries permitted until 2026-07-15 14:34:22.685778339 +0000 UTC m=+35.391854425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs") pod "network-metrics-daemon-9jbvd" (UID: "7d23262f-6be8-46f4-9122-28ce9683ed4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:34:06.685912 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:06.685829 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz podName:7371eff3-e8e5-467d-a2fd-77ebbb193507 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:22.685812524 +0000 UTC m=+35.391888627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwntz" (UniqueName: "kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz") pod "network-check-target-5rtjv" (UID: "7371eff3-e8e5-467d-a2fd-77ebbb193507") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:34:07.077418 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:07.077381 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:07.077418 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:07.077395 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:07.077627 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:07.077515 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:34:07.077690 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:07.077647 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:34:08.077673 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:08.077638 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:08.078113 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:08.077770 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:34:09.079217 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:09.079062 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:09.079217 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:09.079187 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:34:09.079756 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:09.079245 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:09.079756 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:09.079359 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:34:10.077166 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:10.077141 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:10.077320 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:10.077240 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:34:11.077985 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.077559 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:11.078565 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.077615 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:11.078565 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:11.078109 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:34:11.078565 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:11.078147 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:34:11.162406 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.162383 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 14:34:11.162774 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.162753 2574 generic.go:356] "Generic (PLEG): container finished" podID="ee8a402a-6831-434d-b19d-d0a8617c5c91" containerID="d39233b26a214ad34df16a618fb1754e7bcfd36a6ea4993ee48b94515f780956" exitCode=1 Jul 15 14:34:11.162900 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.162817 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" event={"ID":"ee8a402a-6831-434d-b19d-d0a8617c5c91","Type":"ContainerStarted","Data":"727aef59a4772e683cde7358803d218651c12738cde01ac437c42de4bbb46519"} Jul 15 14:34:11.162944 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.162904 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" event={"ID":"ee8a402a-6831-434d-b19d-d0a8617c5c91","Type":"ContainerStarted","Data":"02fc2554823569d0eec5766bdaf88476aea3f8f62c6beaa25162e22fb17cc68c"} Jul 15 14:34:11.162986 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.162947 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" event={"ID":"ee8a402a-6831-434d-b19d-d0a8617c5c91","Type":"ContainerStarted","Data":"5e8da6fedd07c096d57e8e3c9f55090b4cd972f2d37cec0cd09d8964bf3672bc"} Jul 15 14:34:11.162986 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.162982 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" event={"ID":"ee8a402a-6831-434d-b19d-d0a8617c5c91","Type":"ContainerStarted","Data":"08b29f0b27f77c2b2de81ddb3c63cfdad7aeda6e88be0d73306a7347ed9a9b5a"} Jul 15 14:34:11.163086 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.163074 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" event={"ID":"ee8a402a-6831-434d-b19d-d0a8617c5c91","Type":"ContainerDied","Data":"d39233b26a214ad34df16a618fb1754e7bcfd36a6ea4993ee48b94515f780956"} Jul 15 14:34:11.163130 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.163116 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" event={"ID":"ee8a402a-6831-434d-b19d-d0a8617c5c91","Type":"ContainerStarted","Data":"bb53aedaf4a5a307adf3dd3557a4bc213ff8ba3bc76cb029291bfd3dcc8adb02"} Jul 15 14:34:11.164293 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.164268 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6sxqv" event={"ID":"50bab32c-4edf-4ebe-9bd0-cfa9ef0df269","Type":"ContainerStarted","Data":"fcf554de5baac220909754c3278a31b89e9b4d45b20421a5dbb68a6c1713a250"} Jul 15 14:34:11.166138 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.166117 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal" event={"ID":"a8ba0b7cefb501e9fe51d83336aec9d4","Type":"ContainerStarted","Data":"fda40e98d9c6394a9affe98f0f64cb0c268e2d7b637edd06be613f56026776ef"} Jul 15 14:34:11.167421 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.167392 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4hzj7" event={"ID":"0ea1b539-6ec9-41b2-91aa-c27772f97451","Type":"ContainerStarted","Data":"e3c775b3640f2d6d83335904e152128ab0d5ae7dbc44de095552542f8bc5acd3"} Jul 15 14:34:11.168792 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.168770 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" event={"ID":"d017a037-beb6-4045-a169-75f928ba4cae","Type":"ContainerStarted","Data":"555507d1a54752352405bb558642b92bacfe3b3bb1e478d13484c5cf64c3d49b"} Jul 15 14:34:11.170135 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.170113 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" event={"ID":"e72acbe4-3923-4eae-8a56-0bb829dfebf1","Type":"ContainerStarted","Data":"23a8be40bbffb578c5e3bddb45c80bea185010d7f5978434225be40f5f15450a"} Jul 15 14:34:11.171465 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.171443 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5cdr7" event={"ID":"ac545d6a-d123-455e-80d4-890435b906a7","Type":"ContainerStarted","Data":"c7211e1f7c5e8d7c4699eced46ac318e97c84acb27d60ff62db879df0bfd6244"} Jul 15 14:34:11.172928 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.172904 2574 generic.go:356] "Generic (PLEG): container finished" podID="960e795d-fb41-4f9d-8b1f-533f625dbb08" containerID="c9c3ede6eed30891d975ee850481e150806d4789ec3f72a4dea5640cbc163213" exitCode=0 Jul 15 14:34:11.173026 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.172981 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cd8gf" event={"ID":"960e795d-fb41-4f9d-8b1f-533f625dbb08","Type":"ContainerDied","Data":"c9c3ede6eed30891d975ee850481e150806d4789ec3f72a4dea5640cbc163213"} Jul 15 14:34:11.174460 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.174443 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-djz58" event={"ID":"5c59c38d-2211-4547-80df-7505bb4ef76d","Type":"ContainerStarted","Data":"1e701e0d47a893b7f5e63e7d7c47730fc8cd0b30cae2e45c6ba03eeb9005ef7c"} Jul 15 14:34:11.185488 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.185444 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6sxqv" podStartSLOduration=2.899809882 podStartE2EDuration="22.185432232s" totalImagesPullingTime="19.28562235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:11.184937014 +0000 UTC m=+23.891013118" watchObservedRunningTime="2026-07-15 14:34:11.185432232 +0000 UTC m=+23.891508337" Jul 15 14:34:11.215640 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.215592 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4hzj7" podStartSLOduration=2.819575211 podStartE2EDuration="22.215578697s" totalImagesPullingTime="19.396003486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:11.215279496 +0000 UTC m=+23.921355600" watchObservedRunningTime="2026-07-15 14:34:11.215578697 +0000 UTC m=+23.921654803" Jul 15 14:34:11.237961 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.237921 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5cdr7" podStartSLOduration=3.391834379 podStartE2EDuration="22.237906374s" totalImagesPullingTime="18.846071995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:11.237747485 +0000 UTC m=+23.943823590" watchObservedRunningTime="2026-07-15 14:34:11.237906374 +0000 UTC m=+23.943982475" Jul 15 14:34:11.263396 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.263353 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-83.ec2.internal" podStartSLOduration=21.26333637 podStartE2EDuration="21.26333637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:11.262997653 +0000 UTC m=+23.969073757" watchObservedRunningTime="2026-07-15 14:34:11.26333637 +0000 UTC m=+23.969412476" Jul 15 14:34:11.347898 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:11.347830 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kbhnj" podStartSLOduration=3.075602621 podStartE2EDuration="22.347814873s" totalImagesPullingTime="19.272212252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:11.289431694 +0000 UTC m=+23.995507798" watchObservedRunningTime="2026-07-15 14:34:11.347814873 +0000 UTC m=+24.053890977" Jul 15 14:34:12.071188 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:12.071160 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 14:34:12.078230 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:12.078206 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:12.078519 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:12.078367 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:34:12.178940 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:12.178846 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tnwkn" event={"ID":"d7bcb32a-c6a9-4ca7-9cf7-98ad6ec1fea1","Type":"ContainerStarted","Data":"40dae19aaee25b0daf1941227868c32cf9fe376a2bfa984e70afded4e60c9963"} Jul 15 14:34:12.180869 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:12.180838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" event={"ID":"e72acbe4-3923-4eae-8a56-0bb829dfebf1","Type":"ContainerStarted","Data":"e3a0803ae7a5f88c8fa6b9969387a3e4dbe069db4a276e3f25c61c08fe0ff232"} Jul 15 14:34:12.203026 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:12.202986 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-djz58" podStartSLOduration=8.488427229 podStartE2EDuration="23.202969424s" totalImagesPullingTime="14.714542195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:11.38094224 +0000 UTC m=+24.087018359" watchObservedRunningTime="2026-07-15 14:34:12.202969424 +0000 UTC m=+24.909045530" Jul 15 14:34:12.203140 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:12.203088 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tnwkn" podStartSLOduration=3.959831595 podStartE2EDuration="23.203083508s" totalImagesPullingTime="19.243251913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:12.202646394 +0000 UTC m=+24.908722502" watchObservedRunningTime="2026-07-15 14:34:12.203083508 +0000 UTC m=+24.909159641" Jul 15 14:34:12.903658 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:12.903538 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T14:34:12.071183131Z","UUID":"7667c128-b7c2-455e-acbd-52559cc7f4c9","Handler":null,"Name":"","Endpoint":""} Jul 15 14:34:12.905348 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:12.905314 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 14:34:12.905479 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:12.905359 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 14:34:13.078246 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:13.078210 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:13.078679 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:13.078210 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:13.078679 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:13.078340 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:34:13.078679 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:13.078416 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:34:13.180880 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:13.180809 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5cdr7" Jul 15 14:34:13.181435 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:13.181405 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5cdr7" Jul 15 14:34:13.182847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:13.182821 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5cdr7" Jul 15 14:34:13.183266 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:13.183249 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5cdr7" Jul 15 14:34:14.078275 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:14.078079 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:14.078682 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:14.078369 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:34:14.187930 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:14.187902 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 14:34:14.188351 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:14.188323 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" event={"ID":"ee8a402a-6831-434d-b19d-d0a8617c5c91","Type":"ContainerStarted","Data":"4e4a11269969911972586773d79908d138db65189d1b3faaa6d57dbb4674d9d8"} Jul 15 14:34:14.190672 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:14.190602 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" event={"ID":"e72acbe4-3923-4eae-8a56-0bb829dfebf1","Type":"ContainerStarted","Data":"28c51761b5ab15feb62cec10175a9bbd24df054765ac5b125bfda873d84dbf66"} Jul 15 14:34:14.220634 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:14.220594 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2r88f" podStartSLOduration=7.957109877 podStartE2EDuration="25.220579377s" totalImagesPullingTime="17.2634695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:14.220085738 +0000 UTC m=+26.926161843" watchObservedRunningTime="2026-07-15 14:34:14.220579377 +0000 UTC m=+26.926655482" Jul 15 14:34:15.078020 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:15.077987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:15.078209 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:15.077987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:15.078209 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:15.078145 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:34:15.078315 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:15.078206 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:34:16.077674 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:16.077504 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:16.077839 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:16.077737 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:34:16.199251 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:16.199219 2574 generic.go:356] "Generic (PLEG): container finished" podID="960e795d-fb41-4f9d-8b1f-533f625dbb08" containerID="b28e2f9db3333f6d9d6b0f363f1a268658a07d9b60fa5d805211f417010d36d0" exitCode=0 Jul 15 14:34:16.199976 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:16.199305 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cd8gf" event={"ID":"960e795d-fb41-4f9d-8b1f-533f625dbb08","Type":"ContainerDied","Data":"b28e2f9db3333f6d9d6b0f363f1a268658a07d9b60fa5d805211f417010d36d0"} Jul 15 14:34:16.203151 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:16.203135 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 14:34:16.203460 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:16.203440 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" event={"ID":"ee8a402a-6831-434d-b19d-d0a8617c5c91","Type":"ContainerStarted","Data":"5ac73634726f13877574e3421c55cb3f45d11dab21a2263a5643a38a32bae117"} Jul 15 14:34:16.203665 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:16.203647 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:34:16.203750 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:16.203733 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:34:16.203908 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:16.203877 2574 scope.go:117] "RemoveContainer" containerID="d39233b26a214ad34df16a618fb1754e7bcfd36a6ea4993ee48b94515f780956" Jul 15 14:34:16.219581 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:16.219556 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:34:17.077685 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:17.077641 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:17.077833 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:17.077692 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:17.077833 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:17.077763 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:34:17.077927 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:17.077870 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:34:17.209566 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:17.209541 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 14:34:17.209970 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:17.209879 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" event={"ID":"ee8a402a-6831-434d-b19d-d0a8617c5c91","Type":"ContainerStarted","Data":"ec16e48cc0b7157efb7f81bce7ce9538ec5d8a46bfda29f688466687b0f0f77d"} Jul 15 14:34:17.210086 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:17.210071 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:34:17.224854 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:17.224823 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:34:17.305597 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:17.305555 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" podStartSLOduration=8.974191825 podStartE2EDuration="28.305539215s" totalImagesPullingTime="19.33134739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:17.257061269 +0000 UTC m=+29.963137376" watchObservedRunningTime="2026-07-15 14:34:17.305539215 +0000 UTC m=+30.011615320" Jul 15 14:34:18.050345 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:18.050315 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9jbvd"] Jul 15 14:34:18.050499 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:18.050430 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:18.050534 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:18.050521 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:34:18.053662 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:18.053632 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5rtjv"] Jul 15 14:34:18.053775 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:18.053736 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:18.053826 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:18.053812 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:34:18.056252 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:18.056228 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rgz47"] Jul 15 14:34:18.056349 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:18.056301 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:18.056387 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:18.056366 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:34:18.214090 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:18.214007 2574 generic.go:356] "Generic (PLEG): container finished" podID="960e795d-fb41-4f9d-8b1f-533f625dbb08" containerID="c9beb7a31b766c30d3df9e88aa600c20aeebe0857d264a6e81c7f11bde2506c6" exitCode=0 Jul 15 14:34:18.214533 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:18.214092 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cd8gf" event={"ID":"960e795d-fb41-4f9d-8b1f-533f625dbb08","Type":"ContainerDied","Data":"c9beb7a31b766c30d3df9e88aa600c20aeebe0857d264a6e81c7f11bde2506c6"} Jul 15 14:34:19.078604 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:19.078430 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:19.078828 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:19.078674 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:34:20.077213 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:20.077186 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:20.077704 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:20.077311 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:34:20.077704 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:20.077359 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:20.077704 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:20.077453 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:34:20.220493 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:20.220466 2574 generic.go:356] "Generic (PLEG): container finished" podID="960e795d-fb41-4f9d-8b1f-533f625dbb08" containerID="30511a77f734a66e2dc514d3e64b550f84d957a4287befeef132097b660568c2" exitCode=0 Jul 15 14:34:20.220649 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:20.220517 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cd8gf" event={"ID":"960e795d-fb41-4f9d-8b1f-533f625dbb08","Type":"ContainerDied","Data":"30511a77f734a66e2dc514d3e64b550f84d957a4287befeef132097b660568c2"} Jul 15 14:34:21.077480 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:21.077446 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:21.078053 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:21.077581 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:34:22.077404 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:22.077363 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:22.077404 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:22.077401 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:22.077951 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:22.077508 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:34:22.077951 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:22.077637 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:34:22.605462 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:22.605421 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:22.605621 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:22.605536 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 14:34:22.605621 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:22.605591 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret podName:e85e4589-15f8-4790-b90f-246692b35897 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:54.605576562 +0000 UTC m=+67.311652645 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret") pod "global-pull-secret-syncer-rgz47" (UID: "e85e4589-15f8-4790-b90f-246692b35897") : object "kube-system"/"original-pull-secret" not registered Jul 15 14:34:22.706215 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:22.706175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs\") pod \"network-metrics-daemon-9jbvd\" (UID: \"7d23262f-6be8-46f4-9122-28ce9683ed4a\") " pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:22.706414 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:22.706240 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwntz\" (UniqueName: \"kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz\") pod \"network-check-target-5rtjv\" (UID: \"7371eff3-e8e5-467d-a2fd-77ebbb193507\") " pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:22.706414 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:22.706344 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:34:22.706414 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:22.706383 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 14:34:22.706414 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:22.706399 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 14:34:22.706414 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:22.706408 2574 projected.go:196] Error preparing data for projected volume kube-api-access-vwntz for pod openshift-network-diagnostics/network-check-target-5rtjv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:34:22.706414 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:22.706417 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs podName:7d23262f-6be8-46f4-9122-28ce9683ed4a nodeName:}" failed. No retries permitted until 2026-07-15 14:34:54.706400507 +0000 UTC m=+67.412476592 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs") pod "network-metrics-daemon-9jbvd" (UID: "7d23262f-6be8-46f4-9122-28ce9683ed4a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 14:34:22.706720 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:22.706445 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz podName:7371eff3-e8e5-467d-a2fd-77ebbb193507 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:54.70643376 +0000 UTC m=+67.412509843 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vwntz" (UniqueName: "kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz") pod "network-check-target-5rtjv" (UID: "7371eff3-e8e5-467d-a2fd-77ebbb193507") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 14:34:22.788501 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:22.788478 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6sxqv_50bab32c-4edf-4ebe-9bd0-cfa9ef0df269/dns-node-resolver/0.log" Jul 15 14:34:23.078138 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:23.078113 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:23.078724 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:23.078229 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5rtjv" podUID="7371eff3-e8e5-467d-a2fd-77ebbb193507" Jul 15 14:34:23.756949 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:23.756921 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-djz58_5c59c38d-2211-4547-80df-7505bb4ef76d/node-ca/0.log" Jul 15 14:34:24.077424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.077389 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:24.077600 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.077395 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:24.077600 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.077524 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rgz47" podUID="e85e4589-15f8-4790-b90f-246692b35897" Jul 15 14:34:24.077695 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.077609 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9jbvd" podUID="7d23262f-6be8-46f4-9122-28ce9683ed4a" Jul 15 14:34:24.090398 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.090376 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-83.ec2.internal" event="NodeReady" Jul 15 14:34:24.090742 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.090519 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 14:34:24.138493 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.138467 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-845665db47-qpvlp"] Jul 15 14:34:24.147141 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.147119 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.149792 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.149776 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 15 14:34:24.149905 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.149842 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 15 14:34:24.149996 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.149961 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 15 14:34:24.150110 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.150093 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l2mcw\"" Jul 15 14:34:24.152828 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.152808 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bcjgw"] Jul 15 14:34:24.160319 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.160168 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 15 14:34:24.165386 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.165368 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lzntz"] Jul 15 14:34:24.165493 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.165479 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:24.169023 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.169000 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 14:34:24.169112 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.169012 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 14:34:24.169112 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.169034 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w66fn\"" Jul 15 14:34:24.179808 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.179711 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-845665db47-qpvlp"] Jul 15 14:34:24.179937 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.179820 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.179937 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.179894 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bcjgw"] Jul 15 14:34:24.179937 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.179928 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lzntz"] Jul 15 14:34:24.181746 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.181726 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4fmsb"] Jul 15 14:34:24.183952 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.183936 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 14:34:24.184029 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.184014 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 14:34:24.184446 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.184432 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 14:34:24.184742 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.184725 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5q8j8\"" Jul 15 14:34:24.184820 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.184761 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 14:34:24.202234 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.202216 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4fmsb"] Jul 15 14:34:24.202337 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.202317 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4fmsb" Jul 15 14:34:24.205024 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.205007 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 14:34:24.205160 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.205146 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 14:34:24.205230 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.205168 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j8nlk\"" Jul 15 14:34:24.205230 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.205152 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 14:34:24.318380 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aac9896-1d6f-476c-8a30-39ffb309a3de-tmp-dir\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:24.318505 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cbfdfaf-86e4-4a16-83f5-fa3cc2e19ac5-cert\") pod \"ingress-canary-4fmsb\" (UID: \"3cbfdfaf-86e4-4a16-83f5-fa3cc2e19ac5\") " pod="openshift-ingress-canary/ingress-canary-4fmsb" Jul 15 14:34:24.318505 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318408 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-image-registry-private-configuration\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.318505 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318432 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5454a5b-2852-4d93-b072-4d3b7756974d-crio-socket\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.318505 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318485 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.318640 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318521 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cct7c\" (UniqueName: \"kubernetes.io/projected/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-api-access-cct7c\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.318640 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318561 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2aac9896-1d6f-476c-8a30-39ffb309a3de-config-volume\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:24.318640 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318583 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-certificates\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.318640 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318608 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-installation-pull-secrets\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.318640 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318630 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-trusted-ca\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.318782 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318647 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-ca-trust-extracted\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.318782 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318666 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-bound-sa-token\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.318782 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318701 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.318782 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318726 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg4bp\" (UniqueName: \"kubernetes.io/projected/2aac9896-1d6f-476c-8a30-39ffb309a3de-kube-api-access-hg4bp\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:24.318782 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318747 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzb9b\" (UniqueName: \"kubernetes.io/projected/3cbfdfaf-86e4-4a16-83f5-fa3cc2e19ac5-kube-api-access-tzb9b\") pod \"ingress-canary-4fmsb\" (UID: \"3cbfdfaf-86e4-4a16-83f5-fa3cc2e19ac5\") " pod="openshift-ingress-canary/ingress-canary-4fmsb" Jul 15 14:34:24.318782 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.318782 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318779 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pgn5\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-kube-api-access-4pgn5\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.319037 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318822 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5454a5b-2852-4d93-b072-4d3b7756974d-data-volume\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.319037 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.318845 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:24.419413 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aac9896-1d6f-476c-8a30-39ffb309a3de-tmp-dir\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:24.419529 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419423 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cbfdfaf-86e4-4a16-83f5-fa3cc2e19ac5-cert\") pod \"ingress-canary-4fmsb\" (UID: \"3cbfdfaf-86e4-4a16-83f5-fa3cc2e19ac5\") " pod="openshift-ingress-canary/ingress-canary-4fmsb" Jul 15 14:34:24.419529 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419441 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-image-registry-private-configuration\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.419529 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419467 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5454a5b-2852-4d93-b072-4d3b7756974d-crio-socket\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.419529 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419494 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.419529 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419517 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cct7c\" (UniqueName: \"kubernetes.io/projected/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-api-access-cct7c\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.419764 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.419619 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 14:34:24.419764 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419682 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5454a5b-2852-4d93-b072-4d3b7756974d-crio-socket\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.419764 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.419696 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls podName:d5454a5b-2852-4d93-b072-4d3b7756974d nodeName:}" failed. No retries permitted until 2026-07-15 14:34:24.919674039 +0000 UTC m=+37.625750143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lzntz" (UID: "d5454a5b-2852-4d93-b072-4d3b7756974d") : secret "insights-runtime-extractor-tls" not found Jul 15 14:34:24.419764 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419734 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2aac9896-1d6f-476c-8a30-39ffb309a3de-config-volume\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:24.420028 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-certificates\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.420028 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419795 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-installation-pull-secrets\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.420028 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419798 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aac9896-1d6f-476c-8a30-39ffb309a3de-tmp-dir\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:24.420028 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419831 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-trusted-ca\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.420028 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419860 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-ca-trust-extracted\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.420028 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-bound-sa-token\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.420028 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419938 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.420028 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419965 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg4bp\" (UniqueName: \"kubernetes.io/projected/2aac9896-1d6f-476c-8a30-39ffb309a3de-kube-api-access-hg4bp\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:24.420028 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.419991 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzb9b\" (UniqueName: \"kubernetes.io/projected/3cbfdfaf-86e4-4a16-83f5-fa3cc2e19ac5-kube-api-access-tzb9b\") pod \"ingress-canary-4fmsb\" (UID: \"3cbfdfaf-86e4-4a16-83f5-fa3cc2e19ac5\") " pod="openshift-ingress-canary/ingress-canary-4fmsb" Jul 15 14:34:24.420445 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.420035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.420445 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.420076 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pgn5\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-kube-api-access-4pgn5\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.420445 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.420132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5454a5b-2852-4d93-b072-4d3b7756974d-data-volume\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.420445 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.420184 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 14:34:24.420445 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.420190 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:24.420445 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.420260 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-rbac-proxy-cm podName:d5454a5b-2852-4d93-b072-4d3b7756974d nodeName:}" failed. No retries permitted until 2026-07-15 14:34:24.920241464 +0000 UTC m=+37.626317550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lzntz" (UID: "d5454a5b-2852-4d93-b072-4d3b7756974d") : configmap "kube-rbac-proxy" not found Jul 15 14:34:24.420445 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.420308 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 14:34:24.420445 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.420325 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2aac9896-1d6f-476c-8a30-39ffb309a3de-config-volume\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:24.420445 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.420356 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls podName:2aac9896-1d6f-476c-8a30-39ffb309a3de nodeName:}" failed. No retries permitted until 2026-07-15 14:34:24.920340901 +0000 UTC m=+37.626416988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls") pod "dns-default-bcjgw" (UID: "2aac9896-1d6f-476c-8a30-39ffb309a3de") : secret "dns-default-metrics-tls" not found Jul 15 14:34:24.420445 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.420402 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-certificates\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.420951 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.420626 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-ca-trust-extracted\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.420951 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.420732 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 14:34:24.420951 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.420745 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-845665db47-qpvlp: secret "image-registry-tls" not found Jul 15 14:34:24.420951 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.420772 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5454a5b-2852-4d93-b072-4d3b7756974d-data-volume\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.420951 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.420783 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls podName:4bbf6982-e9c9-4e35-9269-d05b4e8a1e98 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:24.920770053 +0000 UTC m=+37.626846153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls") pod "image-registry-845665db47-qpvlp" (UID: "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98") : secret "image-registry-tls" not found Jul 15 14:34:24.421316 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.421297 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-trusted-ca\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.423981 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.423957 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-installation-pull-secrets\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.424089 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.424043 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cbfdfaf-86e4-4a16-83f5-fa3cc2e19ac5-cert\") pod \"ingress-canary-4fmsb\" (UID: \"3cbfdfaf-86e4-4a16-83f5-fa3cc2e19ac5\") " pod="openshift-ingress-canary/ingress-canary-4fmsb" Jul 15 14:34:24.424089 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.424053 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-image-registry-private-configuration\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.438297 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.438263 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cct7c\" (UniqueName: \"kubernetes.io/projected/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-api-access-cct7c\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.439035 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.439010 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pgn5\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-kube-api-access-4pgn5\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.440923 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.440865 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg4bp\" (UniqueName: \"kubernetes.io/projected/2aac9896-1d6f-476c-8a30-39ffb309a3de-kube-api-access-hg4bp\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:24.441910 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.441872 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzb9b\" (UniqueName: \"kubernetes.io/projected/3cbfdfaf-86e4-4a16-83f5-fa3cc2e19ac5-kube-api-access-tzb9b\") pod \"ingress-canary-4fmsb\" (UID: \"3cbfdfaf-86e4-4a16-83f5-fa3cc2e19ac5\") " pod="openshift-ingress-canary/ingress-canary-4fmsb" Jul 15 14:34:24.442496 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.442480 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-bound-sa-token\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.509721 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.509693 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4fmsb" Jul 15 14:34:24.642729 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.642639 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4fmsb"] Jul 15 14:34:24.646709 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:34:24.646679 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbfdfaf_86e4_4a16_83f5_fa3cc2e19ac5.slice/crio-ac35bc7d3d46a55091088d295cde3d100b7826c7153f9772c565949a59c59579 WatchSource:0}: Error finding container ac35bc7d3d46a55091088d295cde3d100b7826c7153f9772c565949a59c59579: Status 404 returned error can't find the container with id ac35bc7d3d46a55091088d295cde3d100b7826c7153f9772c565949a59c59579 Jul 15 14:34:24.923530 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.923439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.923530 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.923484 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:24.923530 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.923505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:24.923805 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.923595 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 14:34:24.923805 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.923597 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 14:34:24.923805 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:24.923641 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:24.923805 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.923598 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 14:34:24.923805 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.923711 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-rbac-proxy-cm podName:d5454a5b-2852-4d93-b072-4d3b7756974d nodeName:}" failed. No retries permitted until 2026-07-15 14:34:25.923690934 +0000 UTC m=+38.629767017 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lzntz" (UID: "d5454a5b-2852-4d93-b072-4d3b7756974d") : configmap "kube-rbac-proxy" not found Jul 15 14:34:24.923805 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.923731 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 14:34:24.923805 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.923734 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls podName:d5454a5b-2852-4d93-b072-4d3b7756974d nodeName:}" failed. No retries permitted until 2026-07-15 14:34:25.923720099 +0000 UTC m=+38.629796181 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lzntz" (UID: "d5454a5b-2852-4d93-b072-4d3b7756974d") : secret "insights-runtime-extractor-tls" not found Jul 15 14:34:24.923805 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.923786 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls podName:2aac9896-1d6f-476c-8a30-39ffb309a3de nodeName:}" failed. No retries permitted until 2026-07-15 14:34:25.923767629 +0000 UTC m=+38.629843718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls") pod "dns-default-bcjgw" (UID: "2aac9896-1d6f-476c-8a30-39ffb309a3de") : secret "dns-default-metrics-tls" not found Jul 15 14:34:24.923805 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.923606 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-845665db47-qpvlp: secret "image-registry-tls" not found Jul 15 14:34:24.924206 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:24.923827 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls podName:4bbf6982-e9c9-4e35-9269-d05b4e8a1e98 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:25.923819713 +0000 UTC m=+38.629895795 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls") pod "image-registry-845665db47-qpvlp" (UID: "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98") : secret "image-registry-tls" not found Jul 15 14:34:25.078014 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:25.077978 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:25.085496 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:25.085471 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 14:34:25.085611 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:25.085473 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 14:34:25.091132 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:25.091115 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m26tp\"" Jul 15 14:34:25.235501 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:25.235376 2574 generic.go:356] "Generic (PLEG): container finished" podID="960e795d-fb41-4f9d-8b1f-533f625dbb08" containerID="1a6651b1b4e3ed30e013810f140711b01b7b114c5489fb1a5e51cbccde5417e6" exitCode=0 Jul 15 14:34:25.235501 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:25.235455 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cd8gf" event={"ID":"960e795d-fb41-4f9d-8b1f-533f625dbb08","Type":"ContainerDied","Data":"1a6651b1b4e3ed30e013810f140711b01b7b114c5489fb1a5e51cbccde5417e6"} Jul 15 14:34:25.237012 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:25.236632 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4fmsb" event={"ID":"3cbfdfaf-86e4-4a16-83f5-fa3cc2e19ac5","Type":"ContainerStarted","Data":"ac35bc7d3d46a55091088d295cde3d100b7826c7153f9772c565949a59c59579"} Jul 15 14:34:25.933977 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:25.933777 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:25.934162 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:25.934001 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:25.934162 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:25.934029 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:25.934162 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:25.933945 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 14:34:25.934162 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:25.934066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:25.934162 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:25.934120 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls podName:d5454a5b-2852-4d93-b072-4d3b7756974d nodeName:}" failed. No retries permitted until 2026-07-15 14:34:27.93409738 +0000 UTC m=+40.640173477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lzntz" (UID: "d5454a5b-2852-4d93-b072-4d3b7756974d") : secret "insights-runtime-extractor-tls" not found Jul 15 14:34:25.934162 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:25.934132 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 14:34:25.934162 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:25.934157 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 14:34:25.934433 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:25.934170 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-845665db47-qpvlp: secret "image-registry-tls" not found Jul 15 14:34:25.934433 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:25.934180 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-rbac-proxy-cm podName:d5454a5b-2852-4d93-b072-4d3b7756974d nodeName:}" failed. No retries permitted until 2026-07-15 14:34:27.934165557 +0000 UTC m=+40.640241640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lzntz" (UID: "d5454a5b-2852-4d93-b072-4d3b7756974d") : configmap "kube-rbac-proxy" not found Jul 15 14:34:25.934433 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:25.934156 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 14:34:25.934433 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:25.934205 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls podName:4bbf6982-e9c9-4e35-9269-d05b4e8a1e98 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:27.934194364 +0000 UTC m=+40.640270462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls") pod "image-registry-845665db47-qpvlp" (UID: "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98") : secret "image-registry-tls" not found Jul 15 14:34:25.934433 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:25.934252 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls podName:2aac9896-1d6f-476c-8a30-39ffb309a3de nodeName:}" failed. No retries permitted until 2026-07-15 14:34:27.934225741 +0000 UTC m=+40.640301838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls") pod "dns-default-bcjgw" (UID: "2aac9896-1d6f-476c-8a30-39ffb309a3de") : secret "dns-default-metrics-tls" not found Jul 15 14:34:26.078184 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:26.078146 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:26.078340 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:26.078146 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:26.085057 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:26.085037 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 14:34:26.085268 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:26.085255 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 14:34:26.086146 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:26.086132 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tz86z\"" Jul 15 14:34:26.242480 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:26.242410 2574 generic.go:356] "Generic (PLEG): container finished" podID="960e795d-fb41-4f9d-8b1f-533f625dbb08" containerID="c5c8048a3fde1222a348299fd5605a215f86f3b9fed9eee41e6c401d86969d78" exitCode=0 Jul 15 14:34:26.242480 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:26.242456 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cd8gf" event={"ID":"960e795d-fb41-4f9d-8b1f-533f625dbb08","Type":"ContainerDied","Data":"c5c8048a3fde1222a348299fd5605a215f86f3b9fed9eee41e6c401d86969d78"} Jul 15 14:34:27.248605 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:27.248551 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cd8gf" event={"ID":"960e795d-fb41-4f9d-8b1f-533f625dbb08","Type":"ContainerStarted","Data":"8c5f4c12bf6ace5d0ac76b3121f3365130c6ba1b4d6d041a8303d00271a91b2a"} Jul 15 14:34:27.951949 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:27.951856 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:27.951949 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:27.951941 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:27.952099 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:27.951964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:27.952099 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:27.952000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:27.952099 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:27.952028 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 14:34:27.952099 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:27.952080 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 14:34:27.952234 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:27.952100 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 14:34:27.952234 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:27.952119 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-845665db47-qpvlp: secret "image-registry-tls" not found Jul 15 14:34:27.952234 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:27.952087 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 14:34:27.952234 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:27.952105 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls podName:d5454a5b-2852-4d93-b072-4d3b7756974d nodeName:}" failed. No retries permitted until 2026-07-15 14:34:31.952083844 +0000 UTC m=+44.658159943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lzntz" (UID: "d5454a5b-2852-4d93-b072-4d3b7756974d") : secret "insights-runtime-extractor-tls" not found Jul 15 14:34:27.952234 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:27.952174 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-rbac-proxy-cm podName:d5454a5b-2852-4d93-b072-4d3b7756974d nodeName:}" failed. No retries permitted until 2026-07-15 14:34:31.952153082 +0000 UTC m=+44.658229169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lzntz" (UID: "d5454a5b-2852-4d93-b072-4d3b7756974d") : configmap "kube-rbac-proxy" not found Jul 15 14:34:27.952234 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:27.952195 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls podName:4bbf6982-e9c9-4e35-9269-d05b4e8a1e98 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:31.952186346 +0000 UTC m=+44.658262433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls") pod "image-registry-845665db47-qpvlp" (UID: "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98") : secret "image-registry-tls" not found Jul 15 14:34:27.952234 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:27.952212 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls podName:2aac9896-1d6f-476c-8a30-39ffb309a3de nodeName:}" failed. No retries permitted until 2026-07-15 14:34:31.952204543 +0000 UTC m=+44.658280637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls") pod "dns-default-bcjgw" (UID: "2aac9896-1d6f-476c-8a30-39ffb309a3de") : secret "dns-default-metrics-tls" not found Jul 15 14:34:28.252232 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:28.252128 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4fmsb" event={"ID":"3cbfdfaf-86e4-4a16-83f5-fa3cc2e19ac5","Type":"ContainerStarted","Data":"0f086c63ce06a48ae7f1d6da0d235e1b92c1796a5f0f0910b0bcfb66f18f3c27"} Jul 15 14:34:28.280552 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:28.280501 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4fmsb" podStartSLOduration=1.5828910010000001 podStartE2EDuration="4.280483087s" totalImagesPullingTime="2.697592086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:28.280135739 +0000 UTC m=+40.986211845" watchObservedRunningTime="2026-07-15 14:34:28.280483087 +0000 UTC m=+40.986559191" Jul 15 14:34:28.280924 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:28.280869 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cd8gf" podStartSLOduration=9.392027608 podStartE2EDuration="39.280864573s" totalImagesPullingTime="29.888836965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:27.288123951 +0000 UTC m=+39.994200073" watchObservedRunningTime="2026-07-15 14:34:28.280864573 +0000 UTC m=+40.986940677" Jul 15 14:34:31.981745 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:31.981701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:31.982196 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:31.981762 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:31.982196 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:31.981792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:31.982196 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:31.981852 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 14:34:31.982196 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:31.981908 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 14:34:31.982196 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:31.981923 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-845665db47-qpvlp: secret "image-registry-tls" not found Jul 15 14:34:31.982196 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:31.981933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:31.982196 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:31.981972 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls podName:4bbf6982-e9c9-4e35-9269-d05b4e8a1e98 nodeName:}" failed. No retries permitted until 2026-07-15 14:34:39.981954519 +0000 UTC m=+52.688030610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls") pod "image-registry-845665db47-qpvlp" (UID: "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98") : secret "image-registry-tls" not found Jul 15 14:34:31.982196 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:31.982003 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls podName:d5454a5b-2852-4d93-b072-4d3b7756974d nodeName:}" failed. No retries permitted until 2026-07-15 14:34:39.98199201 +0000 UTC m=+52.688068098 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lzntz" (UID: "d5454a5b-2852-4d93-b072-4d3b7756974d") : secret "insights-runtime-extractor-tls" not found Jul 15 14:34:31.982196 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:31.982002 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 14:34:31.982196 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:34:31.982066 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls podName:2aac9896-1d6f-476c-8a30-39ffb309a3de nodeName:}" failed. No retries permitted until 2026-07-15 14:34:39.982053489 +0000 UTC m=+52.688129572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls") pod "dns-default-bcjgw" (UID: "2aac9896-1d6f-476c-8a30-39ffb309a3de") : secret "dns-default-metrics-tls" not found Jul 15 14:34:31.982508 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:31.982249 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5454a5b-2852-4d93-b072-4d3b7756974d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:40.046214 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:40.046175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:40.046773 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:40.046232 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:40.046773 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:40.046280 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:40.049603 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:40.049573 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5454a5b-2852-4d93-b072-4d3b7756974d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lzntz\" (UID: \"d5454a5b-2852-4d93-b072-4d3b7756974d\") " pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:40.049603 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:40.049586 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2aac9896-1d6f-476c-8a30-39ffb309a3de-metrics-tls\") pod \"dns-default-bcjgw\" (UID: \"2aac9896-1d6f-476c-8a30-39ffb309a3de\") " pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:40.049738 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:40.049704 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls\") pod \"image-registry-845665db47-qpvlp\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:40.060611 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:40.060590 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:40.073734 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:40.073710 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:40.088070 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:40.088048 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lzntz" Jul 15 14:34:40.254108 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:40.254079 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-845665db47-qpvlp"] Jul 15 14:34:40.257841 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:34:40.257814 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bbf6982_e9c9_4e35_9269_d05b4e8a1e98.slice/crio-3e8662ebee050bdfd440801acb25d4e5f6ea07d8b47fdb154b3e4346394eec5f WatchSource:0}: Error finding container 3e8662ebee050bdfd440801acb25d4e5f6ea07d8b47fdb154b3e4346394eec5f: Status 404 returned error can't find the container with id 3e8662ebee050bdfd440801acb25d4e5f6ea07d8b47fdb154b3e4346394eec5f Jul 15 14:34:40.271436 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:40.271415 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bcjgw"] Jul 15 14:34:40.272087 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:34:40.272067 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aac9896_1d6f_476c_8a30_39ffb309a3de.slice/crio-80717e6c86950b14a4a95fe66cdbaf369278ac2868a15169f0d92a91ae3a8729 WatchSource:0}: Error finding container 80717e6c86950b14a4a95fe66cdbaf369278ac2868a15169f0d92a91ae3a8729: Status 404 returned error can't find the container with id 80717e6c86950b14a4a95fe66cdbaf369278ac2868a15169f0d92a91ae3a8729 Jul 15 14:34:40.282393 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:40.282370 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bcjgw" event={"ID":"2aac9896-1d6f-476c-8a30-39ffb309a3de","Type":"ContainerStarted","Data":"80717e6c86950b14a4a95fe66cdbaf369278ac2868a15169f0d92a91ae3a8729"} Jul 15 14:34:40.283378 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:40.283358 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-845665db47-qpvlp" event={"ID":"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98","Type":"ContainerStarted","Data":"3e8662ebee050bdfd440801acb25d4e5f6ea07d8b47fdb154b3e4346394eec5f"} Jul 15 14:34:40.286827 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:40.286808 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lzntz"] Jul 15 14:34:40.290150 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:34:40.290129 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5454a5b_2852_4d93_b072_4d3b7756974d.slice/crio-534651547d6c7f0d7ba32ac7cd255c6f37744f1c9589859fd3d46b0ff25fdf9f WatchSource:0}: Error finding container 534651547d6c7f0d7ba32ac7cd255c6f37744f1c9589859fd3d46b0ff25fdf9f: Status 404 returned error can't find the container with id 534651547d6c7f0d7ba32ac7cd255c6f37744f1c9589859fd3d46b0ff25fdf9f Jul 15 14:34:41.288460 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:41.288419 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-845665db47-qpvlp" event={"ID":"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98","Type":"ContainerStarted","Data":"8692b1c3739663bbbec5416db11699131558a8ca1cc869ee85a6636582f56fef"} Jul 15 14:34:41.288991 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:41.288538 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:34:41.289544 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:41.289523 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzntz" event={"ID":"d5454a5b-2852-4d93-b072-4d3b7756974d","Type":"ContainerStarted","Data":"534651547d6c7f0d7ba32ac7cd255c6f37744f1c9589859fd3d46b0ff25fdf9f"} Jul 15 14:34:41.320424 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:41.319912 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-845665db47-qpvlp" podStartSLOduration=57.319875406 podStartE2EDuration="57.319875406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:41.318754967 +0000 UTC m=+54.024831073" watchObservedRunningTime="2026-07-15 14:34:41.319875406 +0000 UTC m=+54.025951512" Jul 15 14:34:43.296990 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:43.296953 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bcjgw" event={"ID":"2aac9896-1d6f-476c-8a30-39ffb309a3de","Type":"ContainerStarted","Data":"689b928e4c3dc2bc9408417b7d17f38539ec7c669008d7b9b94c273467b1ac4b"} Jul 15 14:34:43.297383 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:43.297012 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bcjgw" event={"ID":"2aac9896-1d6f-476c-8a30-39ffb309a3de","Type":"ContainerStarted","Data":"ae317981e10da8d855cdf2b9daf0dfb396efd04e939a3d03425236d4664be85d"} Jul 15 14:34:43.297383 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:43.297111 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:43.320707 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:43.320644 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bcjgw" podStartSLOduration=17.178318174 podStartE2EDuration="19.320627674s" totalImagesPullingTime="2.1423095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:43.31973058 +0000 UTC m=+56.025806686" watchObservedRunningTime="2026-07-15 14:34:43.320627674 +0000 UTC m=+56.026703761" Jul 15 14:34:46.312443 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:46.312408 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzntz" event={"ID":"d5454a5b-2852-4d93-b072-4d3b7756974d","Type":"ContainerStarted","Data":"1f9d527a90ff3d648d34695eee9e3a4e81e1bb22889ffa84d5d0be97912d3573"} Jul 15 14:34:48.863135 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:48.863103 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr"] Jul 15 14:34:48.939165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:48.939127 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr"] Jul 15 14:34:48.939165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:48.939166 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-845665db47-qpvlp"] Jul 15 14:34:48.939373 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:48.939278 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:48.947925 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:48.947669 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 15 14:34:48.953846 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:48.953815 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 15 14:34:48.954544 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:48.954528 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 15 14:34:48.976781 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:48.976751 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 15 14:34:48.976976 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:48.976836 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 15 14:34:49.017824 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.017784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92ef2afd-751b-468d-a1b6-060549a142fb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.017995 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.017832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj6gq\" (UniqueName: \"kubernetes.io/projected/92ef2afd-751b-468d-a1b6-060549a142fb-kube-api-access-bj6gq\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.017995 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.017904 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/92ef2afd-751b-468d-a1b6-060549a142fb-hub\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.017995 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.017924 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/92ef2afd-751b-468d-a1b6-060549a142fb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.017995 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.017956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/92ef2afd-751b-468d-a1b6-060549a142fb-ca\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.017995 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.017986 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ef2afd-751b-468d-a1b6-060549a142fb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.068165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.068126 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 15 14:34:49.068165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.068126 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 15 14:34:49.069076 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.069056 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-5bgbj"] Jul 15 14:34:49.091236 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.091196 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-5bgbj" Jul 15 14:34:49.110304 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.110271 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 14:34:49.110465 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.110307 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 14:34:49.110588 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.110572 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-llmsn\"" Jul 15 14:34:49.118309 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.118234 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92ef2afd-751b-468d-a1b6-060549a142fb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.118309 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.118269 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj6gq\" (UniqueName: \"kubernetes.io/projected/92ef2afd-751b-468d-a1b6-060549a142fb-kube-api-access-bj6gq\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.118450 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.118311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/92ef2afd-751b-468d-a1b6-060549a142fb-hub\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.118450 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.118327 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/92ef2afd-751b-468d-a1b6-060549a142fb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.118450 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.118353 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/92ef2afd-751b-468d-a1b6-060549a142fb-ca\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.118450 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.118387 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ef2afd-751b-468d-a1b6-060549a142fb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.119248 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.119208 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ef2afd-751b-468d-a1b6-060549a142fb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.120994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.120961 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/92ef2afd-751b-468d-a1b6-060549a142fb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.121105 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.120975 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92ef2afd-751b-468d-a1b6-060549a142fb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.121175 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.121094 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/92ef2afd-751b-468d-a1b6-060549a142fb-ca\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.121175 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.121170 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/92ef2afd-751b-468d-a1b6-060549a142fb-hub\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.144719 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.144685 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-5bgbj"] Jul 15 14:34:49.152140 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.152107 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj6gq\" (UniqueName: \"kubernetes.io/projected/92ef2afd-751b-468d-a1b6-060549a142fb-kube-api-access-bj6gq\") pod \"cluster-proxy-proxy-agent-6bf6556df9-wn4vr\" (UID: \"92ef2afd-751b-468d-a1b6-060549a142fb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.219276 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.219238 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj784\" (UniqueName: \"kubernetes.io/projected/996f65dd-80ac-421a-99c4-8a9dcfaf287b-kube-api-access-xj784\") pod \"downloads-68c9cbb497-5bgbj\" (UID: \"996f65dd-80ac-421a-99c4-8a9dcfaf287b\") " pod="openshift-console/downloads-68c9cbb497-5bgbj" Jul 15 14:34:49.230985 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.230949 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bdhfs" Jul 15 14:34:49.255840 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.255806 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" Jul 15 14:34:49.320807 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.320773 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj784\" (UniqueName: \"kubernetes.io/projected/996f65dd-80ac-421a-99c4-8a9dcfaf287b-kube-api-access-xj784\") pod \"downloads-68c9cbb497-5bgbj\" (UID: \"996f65dd-80ac-421a-99c4-8a9dcfaf287b\") " pod="openshift-console/downloads-68c9cbb497-5bgbj" Jul 15 14:34:49.339146 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.339119 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj784\" (UniqueName: \"kubernetes.io/projected/996f65dd-80ac-421a-99c4-8a9dcfaf287b-kube-api-access-xj784\") pod \"downloads-68c9cbb497-5bgbj\" (UID: \"996f65dd-80ac-421a-99c4-8a9dcfaf287b\") " pod="openshift-console/downloads-68c9cbb497-5bgbj" Jul 15 14:34:49.399919 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.399803 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-5bgbj" Jul 15 14:34:49.891805 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.891767 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr"] Jul 15 14:34:49.895579 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:34:49.895553 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92ef2afd_751b_468d_a1b6_060549a142fb.slice/crio-969d6e27fe1f3ba95545608d0150f27ca4b96be95bac3a0aaa1cc9296265b6e2 WatchSource:0}: Error finding container 969d6e27fe1f3ba95545608d0150f27ca4b96be95bac3a0aaa1cc9296265b6e2: Status 404 returned error can't find the container with id 969d6e27fe1f3ba95545608d0150f27ca4b96be95bac3a0aaa1cc9296265b6e2 Jul 15 14:34:49.910557 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:49.910533 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-5bgbj"] Jul 15 14:34:49.914053 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:34:49.914025 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996f65dd_80ac_421a_99c4_8a9dcfaf287b.slice/crio-b2f94d152b100deaf08d22b0ec60e6e1d560d1de0ac1e632730555b32d364c1d WatchSource:0}: Error finding container b2f94d152b100deaf08d22b0ec60e6e1d560d1de0ac1e632730555b32d364c1d: Status 404 returned error can't find the container with id b2f94d152b100deaf08d22b0ec60e6e1d560d1de0ac1e632730555b32d364c1d Jul 15 14:34:50.328423 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:50.328383 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzntz" event={"ID":"d5454a5b-2852-4d93-b072-4d3b7756974d","Type":"ContainerStarted","Data":"c9163702e0085a9b05434c26bc6d1d4490260684489da3efa020c9b5ccb447b4"} Jul 15 14:34:50.329538 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:50.329513 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-5bgbj" event={"ID":"996f65dd-80ac-421a-99c4-8a9dcfaf287b","Type":"ContainerStarted","Data":"b2f94d152b100deaf08d22b0ec60e6e1d560d1de0ac1e632730555b32d364c1d"} Jul 15 14:34:50.330571 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:50.330549 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" event={"ID":"92ef2afd-751b-468d-a1b6-060549a142fb","Type":"ContainerStarted","Data":"969d6e27fe1f3ba95545608d0150f27ca4b96be95bac3a0aaa1cc9296265b6e2"} Jul 15 14:34:53.303233 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:53.303204 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bcjgw" Jul 15 14:34:54.665085 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:54.665045 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:54.668249 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:54.668225 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 14:34:54.678103 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:54.678071 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e85e4589-15f8-4790-b90f-246692b35897-original-pull-secret\") pod \"global-pull-secret-syncer-rgz47\" (UID: \"e85e4589-15f8-4790-b90f-246692b35897\") " pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:54.766421 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:54.766383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwntz\" (UniqueName: \"kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz\") pod \"network-check-target-5rtjv\" (UID: \"7371eff3-e8e5-467d-a2fd-77ebbb193507\") " pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:54.766582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:54.766456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs\") pod \"network-metrics-daemon-9jbvd\" (UID: \"7d23262f-6be8-46f4-9122-28ce9683ed4a\") " pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:54.773035 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:54.773003 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 14:34:54.773286 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:54.773270 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 14:34:54.779149 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:54.779126 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d23262f-6be8-46f4-9122-28ce9683ed4a-metrics-certs\") pod \"network-metrics-daemon-9jbvd\" (UID: \"7d23262f-6be8-46f4-9122-28ce9683ed4a\") " pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:54.781158 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:54.781138 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 14:34:54.789972 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:54.789951 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwntz\" (UniqueName: \"kubernetes.io/projected/7371eff3-e8e5-467d-a2fd-77ebbb193507-kube-api-access-vwntz\") pod \"network-check-target-5rtjv\" (UID: \"7371eff3-e8e5-467d-a2fd-77ebbb193507\") " pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:54.794990 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:54.794969 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m26tp\"" Jul 15 14:34:54.800119 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:54.800102 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:34:54.887726 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:54.887690 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rgz47" Jul 15 14:34:54.895295 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:54.895264 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tz86z\"" Jul 15 14:34:54.902063 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:54.902036 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9jbvd" Jul 15 14:34:56.521041 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:56.520989 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9jbvd"] Jul 15 14:34:56.522054 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:56.521763 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rgz47"] Jul 15 14:34:56.525404 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:34:56.525367 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85e4589_15f8_4790_b90f_246692b35897.slice/crio-437a1681b10c814e6bb2d1a6f0c09b3be5e679d969cba1388698e480a69cc13e WatchSource:0}: Error finding container 437a1681b10c814e6bb2d1a6f0c09b3be5e679d969cba1388698e480a69cc13e: Status 404 returned error can't find the container with id 437a1681b10c814e6bb2d1a6f0c09b3be5e679d969cba1388698e480a69cc13e Jul 15 14:34:56.526765 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:34:56.526747 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d23262f_6be8_46f4_9122_28ce9683ed4a.slice/crio-3b129fd3eb41256889e05d1d67bdbd67ee6b394b57c5a919c564577e706f6585 WatchSource:0}: Error finding container 3b129fd3eb41256889e05d1d67bdbd67ee6b394b57c5a919c564577e706f6585: Status 404 returned error can't find the container with id 3b129fd3eb41256889e05d1d67bdbd67ee6b394b57c5a919c564577e706f6585 Jul 15 14:34:56.546344 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:56.546246 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5rtjv"] Jul 15 14:34:56.555123 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:34:56.555094 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7371eff3_e8e5_467d_a2fd_77ebbb193507.slice/crio-3336931183b31a2a2f79c37fa1605e376d66792268a6459442bd2790ba7358a3 WatchSource:0}: Error finding container 3336931183b31a2a2f79c37fa1605e376d66792268a6459442bd2790ba7358a3: Status 404 returned error can't find the container with id 3336931183b31a2a2f79c37fa1605e376d66792268a6459442bd2790ba7358a3 Jul 15 14:34:57.360776 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:57.360742 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rgz47" event={"ID":"e85e4589-15f8-4790-b90f-246692b35897","Type":"ContainerStarted","Data":"437a1681b10c814e6bb2d1a6f0c09b3be5e679d969cba1388698e480a69cc13e"} Jul 15 14:34:57.363931 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:57.363899 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lzntz" event={"ID":"d5454a5b-2852-4d93-b072-4d3b7756974d","Type":"ContainerStarted","Data":"562d487e54a20a5fa761a7089032d47de80dfdb35083b98392258bbb51e87d32"} Jul 15 14:34:57.367374 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:57.367333 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" event={"ID":"92ef2afd-751b-468d-a1b6-060549a142fb","Type":"ContainerStarted","Data":"5df9e6a71f10eb28ad43f34d5226fc1572dd2bb0275980ddbcd1426ad8b7452e"} Jul 15 14:34:57.367482 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:57.367387 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" event={"ID":"92ef2afd-751b-468d-a1b6-060549a142fb","Type":"ContainerStarted","Data":"aba56048fb6b52435c7ccab5de74bfdb28373fb4caa1b410e2a8d06f1d53f4f8"} Jul 15 14:34:57.367482 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:57.367405 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" event={"ID":"92ef2afd-751b-468d-a1b6-060549a142fb","Type":"ContainerStarted","Data":"c02049462f2a596507693b3d8af9174e16e1f78f680a987426264bd075998be6"} Jul 15 14:34:57.369288 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:57.369201 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5rtjv" event={"ID":"7371eff3-e8e5-467d-a2fd-77ebbb193507","Type":"ContainerStarted","Data":"3336931183b31a2a2f79c37fa1605e376d66792268a6459442bd2790ba7358a3"} Jul 15 14:34:57.370846 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:57.370776 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9jbvd" event={"ID":"7d23262f-6be8-46f4-9122-28ce9683ed4a","Type":"ContainerStarted","Data":"3b129fd3eb41256889e05d1d67bdbd67ee6b394b57c5a919c564577e706f6585"} Jul 15 14:34:57.432209 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:57.432140 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6bf6556df9-wn4vr" podStartSLOduration=2.97964656 podStartE2EDuration="9.432118014s" totalImagesPullingTime="6.452471454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:57.431017661 +0000 UTC m=+70.137093767" watchObservedRunningTime="2026-07-15 14:34:57.432118014 +0000 UTC m=+70.138194122" Jul 15 14:34:57.432515 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:57.432478 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lzntz" podStartSLOduration=17.533707009 podStartE2EDuration="33.432469051s" totalImagesPullingTime="15.898762042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:34:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:57.396623707 +0000 UTC m=+70.102699816" watchObservedRunningTime="2026-07-15 14:34:57.432469051 +0000 UTC m=+70.138545156" Jul 15 14:34:58.378006 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:58.377946 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9jbvd" event={"ID":"7d23262f-6be8-46f4-9122-28ce9683ed4a","Type":"ContainerStarted","Data":"009137def16551fd8241dcbfeae87a65292dc8cb2a90504588d22390ad18333c"} Jul 15 14:34:58.378460 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:58.378016 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9jbvd" event={"ID":"7d23262f-6be8-46f4-9122-28ce9683ed4a","Type":"ContainerStarted","Data":"afe964fe7e844764201eb110acd4d134f3495a8d242d84f6f9e5d5e35fc59703"} Jul 15 14:34:58.399695 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:58.399086 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9jbvd" podStartSLOduration=68.468555881 podStartE2EDuration="1m9.399063068s" totalImagesPullingTime="930.507187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:34:58.397420158 +0000 UTC m=+71.103496298" watchObservedRunningTime="2026-07-15 14:34:58.399063068 +0000 UTC m=+71.105139176" Jul 15 14:34:58.947551 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:58.947498 2574 patch_prober.go:28] interesting pod/image-registry-845665db47-qpvlp container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 15 14:34:58.947724 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:34:58.947595 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-845665db47-qpvlp" podUID="4bbf6982-e9c9-4e35-9269-d05b4e8a1e98" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 14:35:00.388323 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:00.388285 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5rtjv" event={"ID":"7371eff3-e8e5-467d-a2fd-77ebbb193507","Type":"ContainerStarted","Data":"c585e38416f0bd73d7c4734b2e28bd66e7cfa11bba8d84a61e164a87df644ec2"} Jul 15 14:35:00.388788 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:00.388386 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:35:00.416062 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:00.416001 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5rtjv" podStartSLOduration=67.934929775 podStartE2EDuration="1m11.415975313s" totalImagesPullingTime="3.481045538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:00.41558204 +0000 UTC m=+73.121658176" watchObservedRunningTime="2026-07-15 14:35:00.415975313 +0000 UTC m=+73.122051418" Jul 15 14:35:02.398232 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:02.398194 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rgz47" event={"ID":"e85e4589-15f8-4790-b90f-246692b35897","Type":"ContainerStarted","Data":"168e4f96dbcf6b1aafa96694924245c4616ae1a7732d67b4990b438654d14426"} Jul 15 14:35:02.417875 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:02.417823 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rgz47" podStartSLOduration=67.923512154 podStartE2EDuration="1m13.417802303s" totalImagesPullingTime="5.494290149s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 14:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:02.417283485 +0000 UTC m=+75.123359592" watchObservedRunningTime="2026-07-15 14:35:02.417802303 +0000 UTC m=+75.123878408" Jul 15 14:35:05.093134 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.093095 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-slghk"] Jul 15 14:35:05.185140 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.185105 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-slghk"] Jul 15 14:35:05.185310 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.185153 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qbdml"] Jul 15 14:35:05.185310 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.185252 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" Jul 15 14:35:05.188526 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.188501 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 15 14:35:05.188655 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.188574 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 15 14:35:05.188655 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.188583 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wtw9f\"" Jul 15 14:35:05.189656 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.189634 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 14:35:05.189791 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.189781 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 14:35:05.189865 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.189785 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 14:35:05.206938 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.206917 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.209813 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.209782 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 14:35:05.210212 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.210028 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 14:35:05.210212 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.210095 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 14:35:05.210212 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.210169 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cdwhh\"" Jul 15 14:35:05.248322 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.248283 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7771d412-bf38-42fb-b668-f8bde1441f12-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-slghk\" (UID: \"7771d412-bf38-42fb-b668-f8bde1441f12\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" Jul 15 14:35:05.248505 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.248335 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7771d412-bf38-42fb-b668-f8bde1441f12-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-slghk\" (UID: \"7771d412-bf38-42fb-b668-f8bde1441f12\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" Jul 15 14:35:05.248505 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.248372 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wm9\" (UniqueName: \"kubernetes.io/projected/7771d412-bf38-42fb-b668-f8bde1441f12-kube-api-access-85wm9\") pod \"openshift-state-metrics-84bd68b896-slghk\" (UID: \"7771d412-bf38-42fb-b668-f8bde1441f12\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" Jul 15 14:35:05.248505 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.248438 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7771d412-bf38-42fb-b668-f8bde1441f12-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-slghk\" (UID: \"7771d412-bf38-42fb-b668-f8bde1441f12\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" Jul 15 14:35:05.349182 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.349090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d6b362ea-1999-46e1-856a-06f61e53fb91-node-exporter-textfile\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.349182 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.349131 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6b362ea-1999-46e1-856a-06f61e53fb91-metrics-client-ca\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.349406 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.349173 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7771d412-bf38-42fb-b668-f8bde1441f12-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-slghk\" (UID: \"7771d412-bf38-42fb-b668-f8bde1441f12\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" Jul 15 14:35:05.349406 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.349231 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d6b362ea-1999-46e1-856a-06f61e53fb91-node-exporter-wtmp\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.349406 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.349258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d6b362ea-1999-46e1-856a-06f61e53fb91-root\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.349406 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.349294 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6b362ea-1999-46e1-856a-06f61e53fb91-sys\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.349406 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.349318 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d6b362ea-1999-46e1-856a-06f61e53fb91-node-exporter-tls\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.349406 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.349343 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7771d412-bf38-42fb-b668-f8bde1441f12-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-slghk\" (UID: \"7771d412-bf38-42fb-b668-f8bde1441f12\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" Jul 15 14:35:05.349406 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.349383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7771d412-bf38-42fb-b668-f8bde1441f12-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-slghk\" (UID: \"7771d412-bf38-42fb-b668-f8bde1441f12\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" Jul 15 14:35:05.349406 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.349407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wm9\" (UniqueName: \"kubernetes.io/projected/7771d412-bf38-42fb-b668-f8bde1441f12-kube-api-access-85wm9\") pod \"openshift-state-metrics-84bd68b896-slghk\" (UID: \"7771d412-bf38-42fb-b668-f8bde1441f12\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" Jul 15 14:35:05.349805 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.349512 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6b362ea-1999-46e1-856a-06f61e53fb91-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.349805 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.349547 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d6b362ea-1999-46e1-856a-06f61e53fb91-node-exporter-accelerators-collector-config\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.349805 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.349574 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m29tg\" (UniqueName: \"kubernetes.io/projected/d6b362ea-1999-46e1-856a-06f61e53fb91-kube-api-access-m29tg\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.350292 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.350264 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7771d412-bf38-42fb-b668-f8bde1441f12-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-slghk\" (UID: \"7771d412-bf38-42fb-b668-f8bde1441f12\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" Jul 15 14:35:05.352327 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.352301 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7771d412-bf38-42fb-b668-f8bde1441f12-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-slghk\" (UID: \"7771d412-bf38-42fb-b668-f8bde1441f12\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" Jul 15 14:35:05.352458 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.352434 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7771d412-bf38-42fb-b668-f8bde1441f12-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-slghk\" (UID: \"7771d412-bf38-42fb-b668-f8bde1441f12\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" Jul 15 14:35:05.360629 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.360590 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wm9\" (UniqueName: \"kubernetes.io/projected/7771d412-bf38-42fb-b668-f8bde1441f12-kube-api-access-85wm9\") pod \"openshift-state-metrics-84bd68b896-slghk\" (UID: \"7771d412-bf38-42fb-b668-f8bde1441f12\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" Jul 15 14:35:05.450146 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.450100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d6b362ea-1999-46e1-856a-06f61e53fb91-node-exporter-textfile\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.450146 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.450156 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6b362ea-1999-46e1-856a-06f61e53fb91-metrics-client-ca\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.450422 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.450208 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d6b362ea-1999-46e1-856a-06f61e53fb91-node-exporter-wtmp\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.450422 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.450240 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d6b362ea-1999-46e1-856a-06f61e53fb91-root\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.450422 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.450270 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6b362ea-1999-46e1-856a-06f61e53fb91-sys\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.450422 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.450292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d6b362ea-1999-46e1-856a-06f61e53fb91-node-exporter-tls\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.450422 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.450345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6b362ea-1999-46e1-856a-06f61e53fb91-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.450422 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.450373 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d6b362ea-1999-46e1-856a-06f61e53fb91-node-exporter-accelerators-collector-config\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.450422 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.450374 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d6b362ea-1999-46e1-856a-06f61e53fb91-node-exporter-wtmp\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.450422 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.450402 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m29tg\" (UniqueName: \"kubernetes.io/projected/d6b362ea-1999-46e1-856a-06f61e53fb91-kube-api-access-m29tg\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.450805 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.450519 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d6b362ea-1999-46e1-856a-06f61e53fb91-node-exporter-textfile\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.450805 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.450591 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d6b362ea-1999-46e1-856a-06f61e53fb91-root\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.450805 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.450635 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6b362ea-1999-46e1-856a-06f61e53fb91-sys\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.451010 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.450979 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6b362ea-1999-46e1-856a-06f61e53fb91-metrics-client-ca\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.451241 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.451218 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d6b362ea-1999-46e1-856a-06f61e53fb91-node-exporter-accelerators-collector-config\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.452981 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.452959 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6b362ea-1999-46e1-856a-06f61e53fb91-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.453081 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.453029 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d6b362ea-1999-46e1-856a-06f61e53fb91-node-exporter-tls\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.461355 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.461325 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m29tg\" (UniqueName: \"kubernetes.io/projected/d6b362ea-1999-46e1-856a-06f61e53fb91-kube-api-access-m29tg\") pod \"node-exporter-qbdml\" (UID: \"d6b362ea-1999-46e1-856a-06f61e53fb91\") " pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:05.496288 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.496248 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" Jul 15 14:35:05.517090 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:05.517053 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qbdml" Jul 15 14:35:06.198382 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.198338 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 14:35:06.278169 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.278126 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 14:35:06.278339 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.278273 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.281982 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.281947 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 14:35:06.282369 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.282345 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 14:35:06.282481 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.282376 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 14:35:06.283103 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.283074 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 14:35:06.283103 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.283101 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 14:35:06.283266 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.283129 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 14:35:06.283266 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.283083 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 14:35:06.283266 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.283076 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 14:35:06.283266 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.283074 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wwqqm\"" Jul 15 14:35:06.283266 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.283260 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 14:35:06.356751 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.356716 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f19dcf-b656-4f23-b019-e62141879968-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.356950 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.356787 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.356950 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.356815 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-config-volume\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.356950 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.356843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.356950 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.356869 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75f19dcf-b656-4f23-b019-e62141879968-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.356950 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.356922 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.357213 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.356956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.357213 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.356988 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nqgm\" (UniqueName: \"kubernetes.io/projected/75f19dcf-b656-4f23-b019-e62141879968-kube-api-access-2nqgm\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.357213 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.357012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f19dcf-b656-4f23-b019-e62141879968-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.357213 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.357032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.357213 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.357095 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75f19dcf-b656-4f23-b019-e62141879968-tls-assets\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.357213 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.357143 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75f19dcf-b656-4f23-b019-e62141879968-config-out\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.357213 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.357172 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-web-config\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.457960 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.457857 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f19dcf-b656-4f23-b019-e62141879968-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.457960 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.457920 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.457960 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.457946 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75f19dcf-b656-4f23-b019-e62141879968-tls-assets\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.458222 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.457984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75f19dcf-b656-4f23-b019-e62141879968-config-out\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.458222 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.458055 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-web-config\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.458222 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:35:06.458071 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/75f19dcf-b656-4f23-b019-e62141879968-alertmanager-trusted-ca-bundle podName:75f19dcf-b656-4f23-b019-e62141879968 nodeName:}" failed. No retries permitted until 2026-07-15 14:35:06.95804236 +0000 UTC m=+79.664118456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/75f19dcf-b656-4f23-b019-e62141879968-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "75f19dcf-b656-4f23-b019-e62141879968") : configmap references non-existent config key: ca-bundle.crt Jul 15 14:35:06.458222 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.458107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f19dcf-b656-4f23-b019-e62141879968-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.458222 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.458175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.458222 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.458202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-config-volume\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.458513 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.458229 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.458513 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.458251 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75f19dcf-b656-4f23-b019-e62141879968-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.458513 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.458278 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.458513 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.458305 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.458513 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.458340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nqgm\" (UniqueName: \"kubernetes.io/projected/75f19dcf-b656-4f23-b019-e62141879968-kube-api-access-2nqgm\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.458754 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.458664 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75f19dcf-b656-4f23-b019-e62141879968-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.459050 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.458994 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f19dcf-b656-4f23-b019-e62141879968-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.462035 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.461982 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.462035 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.462014 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-config-volume\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.462565 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.462398 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.462565 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.462499 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-web-config\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.462809 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.462766 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75f19dcf-b656-4f23-b019-e62141879968-tls-assets\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.462936 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.462858 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75f19dcf-b656-4f23-b019-e62141879968-config-out\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.463175 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.463150 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.463246 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.463203 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.463832 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.463808 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.488089 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.488060 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nqgm\" (UniqueName: \"kubernetes.io/projected/75f19dcf-b656-4f23-b019-e62141879968-kube-api-access-2nqgm\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.962616 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.962566 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f19dcf-b656-4f23-b019-e62141879968-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:06.963519 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:06.963497 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f19dcf-b656-4f23-b019-e62141879968-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:07.189920 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.189867 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:35:07.236063 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.236026 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6668679d49-lzkx5"] Jul 15 14:35:07.243387 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.243355 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.248486 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.248456 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 15 14:35:07.248621 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.248498 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 15 14:35:07.248977 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.248933 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gpkqq\"" Jul 15 14:35:07.249173 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.249149 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5b7618pp4ghsg\"" Jul 15 14:35:07.249875 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.249845 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 15 14:35:07.249987 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.249912 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 15 14:35:07.249987 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.249950 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 15 14:35:07.255634 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.255614 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6668679d49-lzkx5"] Jul 15 14:35:07.366187 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.366152 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.366372 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.366197 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04d62702-a012-4138-85f4-0cf30f0c8aab-metrics-client-ca\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.366372 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.366218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ktr\" (UniqueName: \"kubernetes.io/projected/04d62702-a012-4138-85f4-0cf30f0c8aab-kube-api-access-68ktr\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.366372 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.366280 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.366372 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.366338 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.366593 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.366382 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.366593 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.366434 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-grpc-tls\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.366593 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.366484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-thanos-querier-tls\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.467818 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.467772 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-grpc-tls\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.467818 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.467821 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-thanos-querier-tls\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.468157 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.467863 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.468157 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.467901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04d62702-a012-4138-85f4-0cf30f0c8aab-metrics-client-ca\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.468157 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.468070 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ktr\" (UniqueName: \"kubernetes.io/projected/04d62702-a012-4138-85f4-0cf30f0c8aab-kube-api-access-68ktr\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.468157 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.468108 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.468157 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.468155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.468422 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.468196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.468826 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.468774 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04d62702-a012-4138-85f4-0cf30f0c8aab-metrics-client-ca\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.470995 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.470927 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-grpc-tls\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.471082 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.471000 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.471246 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.471222 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.471320 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.471263 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.471605 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.471584 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.471939 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.471916 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/04d62702-a012-4138-85f4-0cf30f0c8aab-secret-thanos-querier-tls\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.481486 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.481456 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ktr\" (UniqueName: \"kubernetes.io/projected/04d62702-a012-4138-85f4-0cf30f0c8aab-kube-api-access-68ktr\") pod \"thanos-querier-6668679d49-lzkx5\" (UID: \"04d62702-a012-4138-85f4-0cf30f0c8aab\") " pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:07.555472 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:07.555438 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:08.944551 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:08.944518 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:35:09.122061 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:35:09.122012 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b362ea_1999_46e1_856a_06f61e53fb91.slice/crio-7f93b4014095fa7dc3b58fc53ade3799b791283c380f24c5c66c3bff7acd3f2f WatchSource:0}: Error finding container 7f93b4014095fa7dc3b58fc53ade3799b791283c380f24c5c66c3bff7acd3f2f: Status 404 returned error can't find the container with id 7f93b4014095fa7dc3b58fc53ade3799b791283c380f24c5c66c3bff7acd3f2f Jul 15 14:35:09.267165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:09.267144 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-slghk"] Jul 15 14:35:09.269705 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:35:09.269680 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7771d412_bf38_42fb_b668_f8bde1441f12.slice/crio-5ff0b5c5851c7e24b010fd3282c5d7b296e976c2c6b9df6d7503a59c2917b268 WatchSource:0}: Error finding container 5ff0b5c5851c7e24b010fd3282c5d7b296e976c2c6b9df6d7503a59c2917b268: Status 404 returned error can't find the container with id 5ff0b5c5851c7e24b010fd3282c5d7b296e976c2c6b9df6d7503a59c2917b268 Jul 15 14:35:09.423734 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:09.423698 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" event={"ID":"7771d412-bf38-42fb-b668-f8bde1441f12","Type":"ContainerStarted","Data":"5ff0b5c5851c7e24b010fd3282c5d7b296e976c2c6b9df6d7503a59c2917b268"} Jul 15 14:35:09.424683 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:09.424660 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qbdml" event={"ID":"d6b362ea-1999-46e1-856a-06f61e53fb91","Type":"ContainerStarted","Data":"7f93b4014095fa7dc3b58fc53ade3799b791283c380f24c5c66c3bff7acd3f2f"} Jul 15 14:35:09.505362 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:09.505305 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6668679d49-lzkx5"] Jul 15 14:35:09.506404 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:09.506380 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 14:35:09.507070 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:35:09.507045 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d62702_a012_4138_85f4_0cf30f0c8aab.slice/crio-505d435a88dcce2bfdfb4278ee21e49272a1f915b1b9addf983274128182fcfd WatchSource:0}: Error finding container 505d435a88dcce2bfdfb4278ee21e49272a1f915b1b9addf983274128182fcfd: Status 404 returned error can't find the container with id 505d435a88dcce2bfdfb4278ee21e49272a1f915b1b9addf983274128182fcfd Jul 15 14:35:09.507471 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:35:09.507449 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f19dcf_b656_4f23_b019_e62141879968.slice/crio-2dc46dc6d69f9eac6cbce2ea94b466eaa44e769a877c19714eaf73b53130c21f WatchSource:0}: Error finding container 2dc46dc6d69f9eac6cbce2ea94b466eaa44e769a877c19714eaf73b53130c21f: Status 404 returned error can't find the container with id 2dc46dc6d69f9eac6cbce2ea94b466eaa44e769a877c19714eaf73b53130c21f Jul 15 14:35:10.431205 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:10.431159 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-5bgbj" event={"ID":"996f65dd-80ac-421a-99c4-8a9dcfaf287b","Type":"ContainerStarted","Data":"8c1e5de8611dfd7a3459740a1a33e4bfe894429a7e326531fe67f12ad7c93365"} Jul 15 14:35:10.431683 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:10.431353 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-5bgbj" Jul 15 14:35:10.433852 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:10.433817 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" event={"ID":"7771d412-bf38-42fb-b668-f8bde1441f12","Type":"ContainerStarted","Data":"ea1ab86f7a4d32839865a2aba6be9ae0ac41f83dce22a49dcef19e34c8732451"} Jul 15 14:35:10.434003 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:10.433877 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" event={"ID":"7771d412-bf38-42fb-b668-f8bde1441f12","Type":"ContainerStarted","Data":"390d22273d60971ad1c03268b4208e79ed2fdda7925c0a9ea96cbc8a02432696"} Jul 15 14:35:10.435188 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:10.435169 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerStarted","Data":"2dc46dc6d69f9eac6cbce2ea94b466eaa44e769a877c19714eaf73b53130c21f"} Jul 15 14:35:10.436404 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:10.436382 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" event={"ID":"04d62702-a012-4138-85f4-0cf30f0c8aab","Type":"ContainerStarted","Data":"505d435a88dcce2bfdfb4278ee21e49272a1f915b1b9addf983274128182fcfd"} Jul 15 14:35:10.453121 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:10.453092 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-5bgbj" Jul 15 14:35:10.467307 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:10.467257 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-5bgbj" podStartSLOduration=2.896403379 podStartE2EDuration="22.467237548s" totalImagesPullingTime="19.570834169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:10.466800504 +0000 UTC m=+83.172876612" watchObservedRunningTime="2026-07-15 14:35:10.467237548 +0000 UTC m=+83.173313674" Jul 15 14:35:11.444766 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:11.444725 2574 generic.go:356] "Generic (PLEG): container finished" podID="d6b362ea-1999-46e1-856a-06f61e53fb91" containerID="569bc27067ef7746185a644aeafca135a23589c2bab01b6daeab82124181518a" exitCode=0 Jul 15 14:35:11.445317 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:11.444817 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qbdml" event={"ID":"d6b362ea-1999-46e1-856a-06f61e53fb91","Type":"ContainerDied","Data":"569bc27067ef7746185a644aeafca135a23589c2bab01b6daeab82124181518a"} Jul 15 14:35:12.454684 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:12.454611 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qbdml" event={"ID":"d6b362ea-1999-46e1-856a-06f61e53fb91","Type":"ContainerStarted","Data":"f3e648270f3603ab41560b3eab4d322330808e110e943b832bd17160687cd09f"} Jul 15 14:35:12.456978 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:12.456948 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerStarted","Data":"4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81"} Jul 15 14:35:13.462763 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:13.462687 2574 generic.go:356] "Generic (PLEG): container finished" podID="75f19dcf-b656-4f23-b019-e62141879968" containerID="4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81" exitCode=0 Jul 15 14:35:13.463246 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:13.462772 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerDied","Data":"4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81"} Jul 15 14:35:13.465390 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:13.465362 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" event={"ID":"7771d412-bf38-42fb-b668-f8bde1441f12","Type":"ContainerStarted","Data":"e814d4b17f4619930a7679de3d4295e3855fde913ca257065e4fa458e337d933"} Jul 15 14:35:13.467791 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:13.467766 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qbdml" event={"ID":"d6b362ea-1999-46e1-856a-06f61e53fb91","Type":"ContainerStarted","Data":"57d8634aa2c4087a62a20c65e9729c94870eeec467b6adfc3c0bf5e74837733b"} Jul 15 14:35:13.542945 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:13.542815 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-slghk" podStartSLOduration=5.906211983 podStartE2EDuration="8.542724948s" totalImagesPullingTime="2.636512965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:13.538116154 +0000 UTC m=+86.244192259" watchObservedRunningTime="2026-07-15 14:35:13.542724948 +0000 UTC m=+86.248801054" Jul 15 14:35:13.571328 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:13.571265 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qbdml" podStartSLOduration=7.00385483 podStartE2EDuration="8.571241708s" totalImagesPullingTime="1.447350447s" totalInitContainerRuntime="120.036431ms" isStatefulPod=true podCreationTimestamp="2026-07-15 14:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:13.570412483 +0000 UTC m=+86.276488614" watchObservedRunningTime="2026-07-15 14:35:13.571241708 +0000 UTC m=+86.277317814" Jul 15 14:35:13.960900 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:13.960812 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-845665db47-qpvlp" podUID="4bbf6982-e9c9-4e35-9269-d05b4e8a1e98" containerName="registry" containerID="cri-o://8692b1c3739663bbbec5416db11699131558a8ca1cc869ee85a6636582f56fef" gracePeriod=30 Jul 15 14:35:14.251702 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.251620 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:35:14.332683 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.332643 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-certificates\") pod \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " Jul 15 14:35:14.332915 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.332708 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-image-registry-private-configuration\") pod \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " Jul 15 14:35:14.332915 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.332748 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-trusted-ca\") pod \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " Jul 15 14:35:14.332915 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.332773 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-installation-pull-secrets\") pod \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " Jul 15 14:35:14.332915 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.332818 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-bound-sa-token\") pod \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " Jul 15 14:35:14.332915 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.332846 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pgn5\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-kube-api-access-4pgn5\") pod \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " Jul 15 14:35:14.333190 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.332881 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-ca-trust-extracted\") pod \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " Jul 15 14:35:14.333190 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.332974 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls\") pod \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\" (UID: \"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98\") " Jul 15 14:35:14.333282 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.333181 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98" (UID: "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:35:14.333679 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.333565 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98" (UID: "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:35:14.335865 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.335815 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-kube-api-access-4pgn5" (OuterVolumeSpecName: "kube-api-access-4pgn5") pod "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98" (UID: "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98"). InnerVolumeSpecName "kube-api-access-4pgn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:35:14.336200 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.336153 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98" (UID: "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:35:14.336501 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.336466 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98" (UID: "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:35:14.336937 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.336912 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98" (UID: "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:35:14.337014 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.336932 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98" (UID: "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:35:14.345725 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.345687 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98" (UID: "4bbf6982-e9c9-4e35-9269-d05b4e8a1e98"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:35:14.434472 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.434431 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-image-registry-private-configuration\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:35:14.434472 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.434473 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-trusted-ca\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:35:14.434704 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.434491 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-installation-pull-secrets\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:35:14.434704 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.434505 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-bound-sa-token\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:35:14.434704 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.434517 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pgn5\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-kube-api-access-4pgn5\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:35:14.434704 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.434528 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-ca-trust-extracted\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:35:14.434704 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.434540 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-tls\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:35:14.434704 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.434549 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98-registry-certificates\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:35:14.473670 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.473630 2574 generic.go:356] "Generic (PLEG): container finished" podID="4bbf6982-e9c9-4e35-9269-d05b4e8a1e98" containerID="8692b1c3739663bbbec5416db11699131558a8ca1cc869ee85a6636582f56fef" exitCode=0 Jul 15 14:35:14.474138 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.473685 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-845665db47-qpvlp" Jul 15 14:35:14.474138 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.473728 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-845665db47-qpvlp" event={"ID":"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98","Type":"ContainerDied","Data":"8692b1c3739663bbbec5416db11699131558a8ca1cc869ee85a6636582f56fef"} Jul 15 14:35:14.474138 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.473815 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-845665db47-qpvlp" event={"ID":"4bbf6982-e9c9-4e35-9269-d05b4e8a1e98","Type":"ContainerDied","Data":"3e8662ebee050bdfd440801acb25d4e5f6ea07d8b47fdb154b3e4346394eec5f"} Jul 15 14:35:14.474138 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.473847 2574 scope.go:117] "RemoveContainer" containerID="8692b1c3739663bbbec5416db11699131558a8ca1cc869ee85a6636582f56fef" Jul 15 14:35:14.476909 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.476859 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" event={"ID":"04d62702-a012-4138-85f4-0cf30f0c8aab","Type":"ContainerStarted","Data":"10ebe6cf7afbdc1645daacd6238b8d88810f0fd7cb313d2f63e26184db488cd5"} Jul 15 14:35:14.477023 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.476966 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" event={"ID":"04d62702-a012-4138-85f4-0cf30f0c8aab","Type":"ContainerStarted","Data":"6824bd37a9a311a07623dfb5354f8b1d7c4d02540a4714da3f206362982e4035"} Jul 15 14:35:14.477023 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.477016 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" event={"ID":"04d62702-a012-4138-85f4-0cf30f0c8aab","Type":"ContainerStarted","Data":"cc995f1947e1f212088e8ed5d232a93c1ef9e76e81714931aac37e91fe9e3130"} Jul 15 14:35:14.484254 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.484231 2574 scope.go:117] "RemoveContainer" containerID="8692b1c3739663bbbec5416db11699131558a8ca1cc869ee85a6636582f56fef" Jul 15 14:35:14.484579 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:35:14.484552 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8692b1c3739663bbbec5416db11699131558a8ca1cc869ee85a6636582f56fef\": container with ID starting with 8692b1c3739663bbbec5416db11699131558a8ca1cc869ee85a6636582f56fef not found: ID does not exist" containerID="8692b1c3739663bbbec5416db11699131558a8ca1cc869ee85a6636582f56fef" Jul 15 14:35:14.484661 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.484586 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8692b1c3739663bbbec5416db11699131558a8ca1cc869ee85a6636582f56fef"} err="failed to get container status \"8692b1c3739663bbbec5416db11699131558a8ca1cc869ee85a6636582f56fef\": rpc error: code = NotFound desc = could not find container \"8692b1c3739663bbbec5416db11699131558a8ca1cc869ee85a6636582f56fef\": container with ID starting with 8692b1c3739663bbbec5416db11699131558a8ca1cc869ee85a6636582f56fef not found: ID does not exist" Jul 15 14:35:14.506691 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.506613 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-845665db47-qpvlp"] Jul 15 14:35:14.510771 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:14.510738 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-845665db47-qpvlp"] Jul 15 14:35:15.084986 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:15.084629 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bbf6982-e9c9-4e35-9269-d05b4e8a1e98" path="/var/lib/kubelet/pods/4bbf6982-e9c9-4e35-9269-d05b4e8a1e98/volumes" Jul 15 14:35:17.497691 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:17.497597 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerStarted","Data":"82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6"} Jul 15 14:35:17.497691 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:17.497686 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerStarted","Data":"2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a"} Jul 15 14:35:17.498232 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:17.497731 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerStarted","Data":"88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d"} Jul 15 14:35:17.498232 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:17.497773 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerStarted","Data":"08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5"} Jul 15 14:35:17.498232 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:17.497811 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerStarted","Data":"bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe"} Jul 15 14:35:17.498232 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:17.497843 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerStarted","Data":"d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e"} Jul 15 14:35:17.501171 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:17.501138 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" event={"ID":"04d62702-a012-4138-85f4-0cf30f0c8aab","Type":"ContainerStarted","Data":"2c187841eb00b71f032e22c5ebb78852f2b821936390d4d07c1ba39c6b077cce"} Jul 15 14:35:17.501300 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:17.501213 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" event={"ID":"04d62702-a012-4138-85f4-0cf30f0c8aab","Type":"ContainerStarted","Data":"54aba0b5c3164da49fb9f2be36544161656ae46031b7d8399523572c75db20fb"} Jul 15 14:35:17.501300 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:17.501255 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" event={"ID":"04d62702-a012-4138-85f4-0cf30f0c8aab","Type":"ContainerStarted","Data":"f7da82f4a708810d067b527eec84d9339f3653c02cee04387c7b9659b080c9fe"} Jul 15 14:35:17.501300 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:17.501293 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:17.544435 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:17.544382 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.895597538 podStartE2EDuration="11.544366669s" totalImagesPullingTime="5.623162316s" totalInitContainerRuntime="25.606815ms" isStatefulPod=false podCreationTimestamp="2026-07-15 14:35:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:17.54405461 +0000 UTC m=+90.250130715" watchObservedRunningTime="2026-07-15 14:35:17.544366669 +0000 UTC m=+90.250442774" Jul 15 14:35:17.586012 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:17.585951 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" podStartSLOduration=4.053057271 podStartE2EDuration="10.585931838s" totalImagesPullingTime="6.532874567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:35:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:35:17.58347747 +0000 UTC m=+90.289553588" watchObservedRunningTime="2026-07-15 14:35:17.585931838 +0000 UTC m=+90.292007943" Jul 15 14:35:18.511873 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:18.511839 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6668679d49-lzkx5" Jul 15 14:35:31.395714 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:35:31.395681 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5rtjv" Jul 15 14:36:25.682808 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:25.682773 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 14:36:25.683325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:25.683082 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="alertmanager" containerID="cri-o://d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e" gracePeriod=120 Jul 15 14:36:25.683325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:25.683155 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="kube-rbac-proxy-metric" containerID="cri-o://2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a" gracePeriod=120 Jul 15 14:36:25.683325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:25.683203 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="prom-label-proxy" containerID="cri-o://82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6" gracePeriod=120 Jul 15 14:36:25.683325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:25.683224 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="kube-rbac-proxy" containerID="cri-o://88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d" gracePeriod=120 Jul 15 14:36:25.683325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:25.683247 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="config-reloader" containerID="cri-o://bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe" gracePeriod=120 Jul 15 14:36:25.683325 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:25.683177 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="kube-rbac-proxy-web" containerID="cri-o://08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5" gracePeriod=120 Jul 15 14:36:26.749215 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:26.749187 2574 generic.go:356] "Generic (PLEG): container finished" podID="75f19dcf-b656-4f23-b019-e62141879968" containerID="82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6" exitCode=0 Jul 15 14:36:26.749215 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:26.749212 2574 generic.go:356] "Generic (PLEG): container finished" podID="75f19dcf-b656-4f23-b019-e62141879968" containerID="88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d" exitCode=0 Jul 15 14:36:26.749215 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:26.749218 2574 generic.go:356] "Generic (PLEG): container finished" podID="75f19dcf-b656-4f23-b019-e62141879968" containerID="bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe" exitCode=0 Jul 15 14:36:26.749549 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:26.749224 2574 generic.go:356] "Generic (PLEG): container finished" podID="75f19dcf-b656-4f23-b019-e62141879968" containerID="d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e" exitCode=0 Jul 15 14:36:26.749549 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:26.749255 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerDied","Data":"82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6"} Jul 15 14:36:26.749549 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:26.749319 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerDied","Data":"88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d"} Jul 15 14:36:26.749549 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:26.749348 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerDied","Data":"bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe"} Jul 15 14:36:26.749549 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:26.749371 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerDied","Data":"d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e"} Jul 15 14:36:26.920165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:26.920139 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:27.080341 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080313 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75f19dcf-b656-4f23-b019-e62141879968-alertmanager-main-db\") pod \"75f19dcf-b656-4f23-b019-e62141879968\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " Jul 15 14:36:27.080341 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080343 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f19dcf-b656-4f23-b019-e62141879968-alertmanager-trusted-ca-bundle\") pod \"75f19dcf-b656-4f23-b019-e62141879968\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " Jul 15 14:36:27.080589 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080363 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75f19dcf-b656-4f23-b019-e62141879968-tls-assets\") pod \"75f19dcf-b656-4f23-b019-e62141879968\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " Jul 15 14:36:27.080589 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080386 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-web-config\") pod \"75f19dcf-b656-4f23-b019-e62141879968\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " Jul 15 14:36:27.080589 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080411 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy-metric\") pod \"75f19dcf-b656-4f23-b019-e62141879968\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " Jul 15 14:36:27.080589 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080445 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-main-tls\") pod \"75f19dcf-b656-4f23-b019-e62141879968\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " Jul 15 14:36:27.080589 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080483 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nqgm\" (UniqueName: \"kubernetes.io/projected/75f19dcf-b656-4f23-b019-e62141879968-kube-api-access-2nqgm\") pod \"75f19dcf-b656-4f23-b019-e62141879968\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " Jul 15 14:36:27.080589 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080513 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy\") pod \"75f19dcf-b656-4f23-b019-e62141879968\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " Jul 15 14:36:27.080589 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080547 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f19dcf-b656-4f23-b019-e62141879968-metrics-client-ca\") pod \"75f19dcf-b656-4f23-b019-e62141879968\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " Jul 15 14:36:27.080589 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080569 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75f19dcf-b656-4f23-b019-e62141879968-config-out\") pod \"75f19dcf-b656-4f23-b019-e62141879968\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " Jul 15 14:36:27.081123 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080598 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-config-volume\") pod \"75f19dcf-b656-4f23-b019-e62141879968\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " Jul 15 14:36:27.081123 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080630 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy-web\") pod \"75f19dcf-b656-4f23-b019-e62141879968\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " Jul 15 14:36:27.081123 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080689 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-cluster-tls-config\") pod \"75f19dcf-b656-4f23-b019-e62141879968\" (UID: \"75f19dcf-b656-4f23-b019-e62141879968\") " Jul 15 14:36:27.081123 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080682 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f19dcf-b656-4f23-b019-e62141879968-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "75f19dcf-b656-4f23-b019-e62141879968" (UID: "75f19dcf-b656-4f23-b019-e62141879968"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:36:27.081123 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080748 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f19dcf-b656-4f23-b019-e62141879968-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "75f19dcf-b656-4f23-b019-e62141879968" (UID: "75f19dcf-b656-4f23-b019-e62141879968"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:36:27.081123 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080934 2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75f19dcf-b656-4f23-b019-e62141879968-alertmanager-main-db\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:36:27.081123 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.080955 2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75f19dcf-b656-4f23-b019-e62141879968-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:36:27.083335 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.083301 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "75f19dcf-b656-4f23-b019-e62141879968" (UID: "75f19dcf-b656-4f23-b019-e62141879968"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:36:27.083901 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.083549 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f19dcf-b656-4f23-b019-e62141879968-config-out" (OuterVolumeSpecName: "config-out") pod "75f19dcf-b656-4f23-b019-e62141879968" (UID: "75f19dcf-b656-4f23-b019-e62141879968"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 14:36:27.083901 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.083621 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "75f19dcf-b656-4f23-b019-e62141879968" (UID: "75f19dcf-b656-4f23-b019-e62141879968"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:36:27.084052 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.083922 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f19dcf-b656-4f23-b019-e62141879968-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "75f19dcf-b656-4f23-b019-e62141879968" (UID: "75f19dcf-b656-4f23-b019-e62141879968"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:36:27.084052 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.083955 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-config-volume" (OuterVolumeSpecName: "config-volume") pod "75f19dcf-b656-4f23-b019-e62141879968" (UID: "75f19dcf-b656-4f23-b019-e62141879968"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:36:27.084052 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.083987 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f19dcf-b656-4f23-b019-e62141879968-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "75f19dcf-b656-4f23-b019-e62141879968" (UID: "75f19dcf-b656-4f23-b019-e62141879968"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 14:36:27.084370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.084344 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f19dcf-b656-4f23-b019-e62141879968-kube-api-access-2nqgm" (OuterVolumeSpecName: "kube-api-access-2nqgm") pod "75f19dcf-b656-4f23-b019-e62141879968" (UID: "75f19dcf-b656-4f23-b019-e62141879968"). InnerVolumeSpecName "kube-api-access-2nqgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:36:27.085853 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.085826 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "75f19dcf-b656-4f23-b019-e62141879968" (UID: "75f19dcf-b656-4f23-b019-e62141879968"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:36:27.086197 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.086173 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "75f19dcf-b656-4f23-b019-e62141879968" (UID: "75f19dcf-b656-4f23-b019-e62141879968"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:36:27.087193 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.087109 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "75f19dcf-b656-4f23-b019-e62141879968" (UID: "75f19dcf-b656-4f23-b019-e62141879968"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:36:27.093462 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.093441 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-web-config" (OuterVolumeSpecName: "web-config") pod "75f19dcf-b656-4f23-b019-e62141879968" (UID: "75f19dcf-b656-4f23-b019-e62141879968"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:36:27.182092 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.182043 2574 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-cluster-tls-config\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:36:27.182092 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.182092 2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75f19dcf-b656-4f23-b019-e62141879968-tls-assets\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:36:27.182092 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.182106 2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-web-config\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:36:27.182342 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.182121 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:36:27.182342 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.182138 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-main-tls\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:36:27.182342 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.182150 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nqgm\" (UniqueName: \"kubernetes.io/projected/75f19dcf-b656-4f23-b019-e62141879968-kube-api-access-2nqgm\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:36:27.182342 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.182162 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:36:27.182342 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.182174 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75f19dcf-b656-4f23-b019-e62141879968-metrics-client-ca\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:36:27.182342 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.182187 2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75f19dcf-b656-4f23-b019-e62141879968-config-out\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:36:27.182342 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.182197 2574 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-config-volume\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:36:27.182342 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.182209 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75f19dcf-b656-4f23-b019-e62141879968-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:36:27.755479 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.755446 2574 generic.go:356] "Generic (PLEG): container finished" podID="75f19dcf-b656-4f23-b019-e62141879968" containerID="2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a" exitCode=0 Jul 15 14:36:27.755479 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.755476 2574 generic.go:356] "Generic (PLEG): container finished" podID="75f19dcf-b656-4f23-b019-e62141879968" containerID="08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5" exitCode=0 Jul 15 14:36:27.755982 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.755522 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:27.755982 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.755530 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerDied","Data":"2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a"} Jul 15 14:36:27.755982 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.755599 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerDied","Data":"08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5"} Jul 15 14:36:27.755982 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.755631 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75f19dcf-b656-4f23-b019-e62141879968","Type":"ContainerDied","Data":"2dc46dc6d69f9eac6cbce2ea94b466eaa44e769a877c19714eaf73b53130c21f"} Jul 15 14:36:27.755982 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.755674 2574 scope.go:117] "RemoveContainer" containerID="82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6" Jul 15 14:36:27.763299 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.763272 2574 scope.go:117] "RemoveContainer" containerID="2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a" Jul 15 14:36:27.771478 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.771462 2574 scope.go:117] "RemoveContainer" containerID="88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d" Jul 15 14:36:27.779130 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.779107 2574 scope.go:117] "RemoveContainer" containerID="08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5" Jul 15 14:36:27.782094 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.782076 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 14:36:27.786839 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.786819 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 14:36:27.788006 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.787993 2574 scope.go:117] "RemoveContainer" containerID="bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe" Jul 15 14:36:27.794661 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.794646 2574 scope.go:117] "RemoveContainer" containerID="d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e" Jul 15 14:36:27.801646 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.801496 2574 scope.go:117] "RemoveContainer" containerID="4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81" Jul 15 14:36:27.807682 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.807632 2574 scope.go:117] "RemoveContainer" containerID="82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6" Jul 15 14:36:27.807910 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:36:27.807879 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6\": container with ID starting with 82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6 not found: ID does not exist" containerID="82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6" Jul 15 14:36:27.807992 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.807923 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6"} err="failed to get container status \"82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6\": rpc error: code = NotFound desc = could not find container \"82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6\": container with ID starting with 82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6 not found: ID does not exist" Jul 15 14:36:27.807992 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.807946 2574 scope.go:117] "RemoveContainer" containerID="2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a" Jul 15 14:36:27.808176 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:36:27.808157 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a\": container with ID starting with 2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a not found: ID does not exist" containerID="2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a" Jul 15 14:36:27.808212 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.808183 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a"} err="failed to get container status \"2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a\": rpc error: code = NotFound desc = could not find container \"2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a\": container with ID starting with 2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a not found: ID does not exist" Jul 15 14:36:27.808212 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.808199 2574 scope.go:117] "RemoveContainer" containerID="88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d" Jul 15 14:36:27.808449 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:36:27.808432 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d\": container with ID starting with 88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d not found: ID does not exist" containerID="88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d" Jul 15 14:36:27.808488 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.808454 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d"} err="failed to get container status \"88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d\": rpc error: code = NotFound desc = could not find container \"88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d\": container with ID starting with 88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d not found: ID does not exist" Jul 15 14:36:27.808488 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.808468 2574 scope.go:117] "RemoveContainer" containerID="08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5" Jul 15 14:36:27.808670 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:36:27.808653 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5\": container with ID starting with 08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5 not found: ID does not exist" containerID="08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5" Jul 15 14:36:27.808707 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.808675 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5"} err="failed to get container status \"08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5\": rpc error: code = NotFound desc = could not find container \"08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5\": container with ID starting with 08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5 not found: ID does not exist" Jul 15 14:36:27.808707 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.808690 2574 scope.go:117] "RemoveContainer" containerID="bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe" Jul 15 14:36:27.808879 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:36:27.808865 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe\": container with ID starting with bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe not found: ID does not exist" containerID="bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe" Jul 15 14:36:27.808934 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.808894 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe"} err="failed to get container status \"bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe\": rpc error: code = NotFound desc = could not find container \"bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe\": container with ID starting with bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe not found: ID does not exist" Jul 15 14:36:27.808934 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.808908 2574 scope.go:117] "RemoveContainer" containerID="d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e" Jul 15 14:36:27.809114 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:36:27.809096 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e\": container with ID starting with d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e not found: ID does not exist" containerID="d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e" Jul 15 14:36:27.809181 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.809123 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e"} err="failed to get container status \"d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e\": rpc error: code = NotFound desc = could not find container \"d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e\": container with ID starting with d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e not found: ID does not exist" Jul 15 14:36:27.809181 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.809145 2574 scope.go:117] "RemoveContainer" containerID="4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81" Jul 15 14:36:27.809410 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:36:27.809392 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81\": container with ID starting with 4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81 not found: ID does not exist" containerID="4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81" Jul 15 14:36:27.809450 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.809415 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81"} err="failed to get container status \"4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81\": rpc error: code = NotFound desc = could not find container \"4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81\": container with ID starting with 4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81 not found: ID does not exist" Jul 15 14:36:27.809450 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.809427 2574 scope.go:117] "RemoveContainer" containerID="82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6" Jul 15 14:36:27.809626 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.809607 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6"} err="failed to get container status \"82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6\": rpc error: code = NotFound desc = could not find container \"82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6\": container with ID starting with 82c3474aeb92e414a28f2ef7febbbf680fdc9aac4e0661881020f9233f77b5c6 not found: ID does not exist" Jul 15 14:36:27.809695 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.809628 2574 scope.go:117] "RemoveContainer" containerID="2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a" Jul 15 14:36:27.809856 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.809837 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a"} err="failed to get container status \"2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a\": rpc error: code = NotFound desc = could not find container \"2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a\": container with ID starting with 2b120658e222ab37344b2049614a8472e4f3ed45f82995309306df1d4be0801a not found: ID does not exist" Jul 15 14:36:27.809929 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.809856 2574 scope.go:117] "RemoveContainer" containerID="88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d" Jul 15 14:36:27.810073 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.810055 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d"} err="failed to get container status \"88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d\": rpc error: code = NotFound desc = could not find container \"88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d\": container with ID starting with 88755833595b840376f812f6c53d88a956c0271bf6a270e27ade7d803ff1fa2d not found: ID does not exist" Jul 15 14:36:27.810119 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.810073 2574 scope.go:117] "RemoveContainer" containerID="08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5" Jul 15 14:36:27.810253 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.810238 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5"} err="failed to get container status \"08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5\": rpc error: code = NotFound desc = could not find container \"08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5\": container with ID starting with 08174d704b0a707a4cc17003ccd85fb0dee0ec0fd12dae81f4ccdc9a53d222b5 not found: ID does not exist" Jul 15 14:36:27.810294 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.810254 2574 scope.go:117] "RemoveContainer" containerID="bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe" Jul 15 14:36:27.810428 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.810399 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe"} err="failed to get container status \"bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe\": rpc error: code = NotFound desc = could not find container \"bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe\": container with ID starting with bc36a8abc89d32727e0dbc2f2403316c73fc2443f8698c5bd79ba0af8cabc8fe not found: ID does not exist" Jul 15 14:36:27.810428 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.810421 2574 scope.go:117] "RemoveContainer" containerID="d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e" Jul 15 14:36:27.810634 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.810618 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e"} err="failed to get container status \"d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e\": rpc error: code = NotFound desc = could not find container \"d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e\": container with ID starting with d4d23b1a87b20dc9c5066dd2ef5e8e05bb4dc26d6f96fd30846108e4f61a061e not found: ID does not exist" Jul 15 14:36:27.810691 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.810633 2574 scope.go:117] "RemoveContainer" containerID="4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81" Jul 15 14:36:27.810815 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.810799 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81"} err="failed to get container status \"4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81\": rpc error: code = NotFound desc = could not find container \"4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81\": container with ID starting with 4a183829e45898ad90f3ccb58330026ad4ca2bfa3c473c66e12600b93a2cad81 not found: ID does not exist" Jul 15 14:36:27.819219 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819200 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 14:36:27.819483 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819468 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="kube-rbac-proxy" Jul 15 14:36:27.819551 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819486 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="kube-rbac-proxy" Jul 15 14:36:27.819551 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819504 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="alertmanager" Jul 15 14:36:27.819551 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819513 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="alertmanager" Jul 15 14:36:27.819551 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819523 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="prom-label-proxy" Jul 15 14:36:27.819551 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819532 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="prom-label-proxy" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819552 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="init-config-reloader" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819561 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="init-config-reloader" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819573 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="config-reloader" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819581 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="config-reloader" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819594 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="kube-rbac-proxy-web" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819602 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="kube-rbac-proxy-web" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819614 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="kube-rbac-proxy-metric" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819623 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="kube-rbac-proxy-metric" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819632 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bbf6982-e9c9-4e35-9269-d05b4e8a1e98" containerName="registry" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819640 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bbf6982-e9c9-4e35-9269-d05b4e8a1e98" containerName="registry" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819699 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="config-reloader" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819710 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="prom-label-proxy" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819720 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="kube-rbac-proxy" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819731 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bbf6982-e9c9-4e35-9269-d05b4e8a1e98" containerName="registry" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819740 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="alertmanager" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819749 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="kube-rbac-proxy-web" Jul 15 14:36:27.819787 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.819757 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f19dcf-b656-4f23-b019-e62141879968" containerName="kube-rbac-proxy-metric" Jul 15 14:36:27.823395 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.823380 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:27.826804 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.826786 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 14:36:27.826871 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.826828 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 14:36:27.827048 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.827030 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 14:36:27.827125 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.827057 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 14:36:27.827219 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.827202 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 14:36:27.827298 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.827282 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 14:36:27.827356 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.827313 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wwqqm\"" Jul 15 14:36:27.827413 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.827380 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 14:36:27.827478 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.827465 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 14:36:27.834362 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.834345 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 14:36:27.841740 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.841717 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 14:36:27.986879 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.986794 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-config-volume\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:27.986879 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.986829 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:27.986879 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.986857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:27.987108 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.986911 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvkxr\" (UniqueName: \"kubernetes.io/projected/409a0c7c-4e42-4c35-b039-76669319eb94-kube-api-access-rvkxr\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:27.987108 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.986934 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/409a0c7c-4e42-4c35-b039-76669319eb94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:27.987108 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.986977 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:27.987108 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.986994 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-web-config\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:27.987108 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.987028 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:27.987108 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.987065 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:27.987108 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.987085 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/409a0c7c-4e42-4c35-b039-76669319eb94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:27.987303 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.987129 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/409a0c7c-4e42-4c35-b039-76669319eb94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:27.987303 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.987167 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/409a0c7c-4e42-4c35-b039-76669319eb94-config-out\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:27.987303 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:27.987190 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/409a0c7c-4e42-4c35-b039-76669319eb94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.087791 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.087758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.087960 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.087809 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.087960 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.087842 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/409a0c7c-4e42-4c35-b039-76669319eb94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.087960 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.087868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/409a0c7c-4e42-4c35-b039-76669319eb94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.087960 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.087928 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/409a0c7c-4e42-4c35-b039-76669319eb94-config-out\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.088160 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.087960 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/409a0c7c-4e42-4c35-b039-76669319eb94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.088160 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.087986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-config-volume\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.088160 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.088012 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.088160 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.088041 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.088160 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.088065 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvkxr\" (UniqueName: \"kubernetes.io/projected/409a0c7c-4e42-4c35-b039-76669319eb94-kube-api-access-rvkxr\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.088160 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.088087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/409a0c7c-4e42-4c35-b039-76669319eb94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.088160 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.088127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.088160 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.088153 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-web-config\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.088534 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.088507 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/409a0c7c-4e42-4c35-b039-76669319eb94-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.089473 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.088800 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/409a0c7c-4e42-4c35-b039-76669319eb94-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.089473 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.089056 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/409a0c7c-4e42-4c35-b039-76669319eb94-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.090807 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.090781 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/409a0c7c-4e42-4c35-b039-76669319eb94-config-out\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.091089 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.091015 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.091287 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.091258 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.091365 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.091322 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.091365 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.091322 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-config-volume\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.091647 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.091624 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/409a0c7c-4e42-4c35-b039-76669319eb94-tls-assets\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.091647 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.091636 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-web-config\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.092031 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.092011 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.092951 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.092935 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/409a0c7c-4e42-4c35-b039-76669319eb94-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.104584 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.101920 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvkxr\" (UniqueName: \"kubernetes.io/projected/409a0c7c-4e42-4c35-b039-76669319eb94-kube-api-access-rvkxr\") pod \"alertmanager-main-0\" (UID: \"409a0c7c-4e42-4c35-b039-76669319eb94\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.132857 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.132817 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 14:36:28.274812 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.274788 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 14:36:28.276558 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:36:28.276533 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409a0c7c_4e42_4c35_b039_76669319eb94.slice/crio-9627ce272cbfa0399817cb0d3a4ba53fab59f6c89a4a76d3116e2e779196b822 WatchSource:0}: Error finding container 9627ce272cbfa0399817cb0d3a4ba53fab59f6c89a4a76d3116e2e779196b822: Status 404 returned error can't find the container with id 9627ce272cbfa0399817cb0d3a4ba53fab59f6c89a4a76d3116e2e779196b822 Jul 15 14:36:28.761454 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.761416 2574 generic.go:356] "Generic (PLEG): container finished" podID="409a0c7c-4e42-4c35-b039-76669319eb94" containerID="8ff1a34bc93a2f056e1e31525678495309e1242fcbdf7a8def306d023b468669" exitCode=0 Jul 15 14:36:28.761909 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.761463 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"409a0c7c-4e42-4c35-b039-76669319eb94","Type":"ContainerDied","Data":"8ff1a34bc93a2f056e1e31525678495309e1242fcbdf7a8def306d023b468669"} Jul 15 14:36:28.761909 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:28.761520 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"409a0c7c-4e42-4c35-b039-76669319eb94","Type":"ContainerStarted","Data":"9627ce272cbfa0399817cb0d3a4ba53fab59f6c89a4a76d3116e2e779196b822"} Jul 15 14:36:29.084526 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.084496 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f19dcf-b656-4f23-b019-e62141879968" path="/var/lib/kubelet/pods/75f19dcf-b656-4f23-b019-e62141879968/volumes" Jul 15 14:36:29.768478 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.768437 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"409a0c7c-4e42-4c35-b039-76669319eb94","Type":"ContainerStarted","Data":"b19b08f05cd239e241cb41ab53d40dc92abe7a20143b0cffa7f343223fdcc611"} Jul 15 14:36:29.768818 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.768519 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"409a0c7c-4e42-4c35-b039-76669319eb94","Type":"ContainerStarted","Data":"be10e6a5400be86627611f624258242a15e177f1081ab09b893548738e937fdc"} Jul 15 14:36:29.768818 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.768559 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"409a0c7c-4e42-4c35-b039-76669319eb94","Type":"ContainerStarted","Data":"dcac6ccfd62ac54a80a7a813766041f2463ae251ade34c527330a1fd7d40ce51"} Jul 15 14:36:29.768818 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.768600 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"409a0c7c-4e42-4c35-b039-76669319eb94","Type":"ContainerStarted","Data":"136693b4256a395269d49dfc542e39122d03daee41a449caf91ef916953857a2"} Jul 15 14:36:29.768818 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.768637 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"409a0c7c-4e42-4c35-b039-76669319eb94","Type":"ContainerStarted","Data":"9fc1f00cfe81ea6e09598fca5b3a60cccbb4571495f3e94c274adc156515c40e"} Jul 15 14:36:29.768818 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.768672 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"409a0c7c-4e42-4c35-b039-76669319eb94","Type":"ContainerStarted","Data":"cc713231fe279074f8d5cec4332ac9273acec8a36a1d203d46f39a77c41cec61"} Jul 15 14:36:29.778705 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.778677 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-65964b6d55-57llg"] Jul 15 14:36:29.780843 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.780827 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:29.786777 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.786755 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-v5cs6\"" Jul 15 14:36:29.786896 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.786756 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 15 14:36:29.786896 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.786807 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 15 14:36:29.787126 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.787106 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 15 14:36:29.787445 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.787424 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 15 14:36:29.788036 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.788020 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 15 14:36:29.797381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.796279 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 15 14:36:29.802681 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.802661 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65964b6d55-57llg"] Jul 15 14:36:29.817100 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.817053 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.802345987 podStartE2EDuration="2.817037324s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.691337ms" isStatefulPod=false podCreationTimestamp="2026-07-15 14:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:36:29.816846482 +0000 UTC m=+162.522922588" watchObservedRunningTime="2026-07-15 14:36:29.817037324 +0000 UTC m=+162.523113443" Jul 15 14:36:29.901902 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.901842 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:29.902094 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.901917 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-telemeter-client-tls\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:29.902094 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.901987 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-federate-client-tls\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:29.902094 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.902014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-metrics-client-ca\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:29.902094 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.902043 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-secret-telemeter-client\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:29.902302 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.902274 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:29.902348 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.902327 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-serving-certs-ca-bundle\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:29.902380 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:29.902365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2swfk\" (UniqueName: \"kubernetes.io/projected/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-kube-api-access-2swfk\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.003426 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.003387 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2swfk\" (UniqueName: \"kubernetes.io/projected/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-kube-api-access-2swfk\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.003538 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.003443 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.003538 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.003463 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-telemeter-client-tls\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.003538 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.003496 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-federate-client-tls\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.003538 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.003514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-metrics-client-ca\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.003697 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.003668 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-secret-telemeter-client\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.003771 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.003751 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.003827 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.003802 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-serving-certs-ca-bundle\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.004381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.004352 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-metrics-client-ca\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.004565 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.004536 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-serving-certs-ca-bundle\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.004666 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.004562 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.006219 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.006192 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.006295 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.006259 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-federate-client-tls\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.006295 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.006290 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-telemeter-client-tls\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.006410 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.006387 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-secret-telemeter-client\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.014287 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.014266 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2swfk\" (UniqueName: \"kubernetes.io/projected/776ec54b-28e9-42b0-aa2f-5a6fa2e3e992-kube-api-access-2swfk\") pod \"telemeter-client-65964b6d55-57llg\" (UID: \"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992\") " pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.089828 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.089802 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" Jul 15 14:36:30.229720 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.229687 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65964b6d55-57llg"] Jul 15 14:36:30.233205 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:36:30.233181 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776ec54b_28e9_42b0_aa2f_5a6fa2e3e992.slice/crio-e9fa031abf998eac3af7fb280b9c40f7a8e6de653e18777f8d0642662957e424 WatchSource:0}: Error finding container e9fa031abf998eac3af7fb280b9c40f7a8e6de653e18777f8d0642662957e424: Status 404 returned error can't find the container with id e9fa031abf998eac3af7fb280b9c40f7a8e6de653e18777f8d0642662957e424 Jul 15 14:36:30.772788 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:30.772756 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" event={"ID":"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992","Type":"ContainerStarted","Data":"e9fa031abf998eac3af7fb280b9c40f7a8e6de653e18777f8d0642662957e424"} Jul 15 14:36:32.781943 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:32.781908 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" event={"ID":"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992","Type":"ContainerStarted","Data":"27fd29a0798ceacae57b9c3528d1bed552f5d734c998a2d00c96a1440f8f4b64"} Jul 15 14:36:32.782355 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:32.781961 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" event={"ID":"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992","Type":"ContainerStarted","Data":"3bdbb07969cf4087cc090ac7548d6c210148ed528aa57a479face4709c936fbe"} Jul 15 14:36:32.782355 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:32.781980 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" event={"ID":"776ec54b-28e9-42b0-aa2f-5a6fa2e3e992","Type":"ContainerStarted","Data":"d9dca439bc4bbadc1eff3bc4455dfc99fbba0c96fc30c02cd7591dfee692f5b3"} Jul 15 14:36:32.811563 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:36:32.811513 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-65964b6d55-57llg" podStartSLOduration=1.87748703 podStartE2EDuration="3.81149684s" totalImagesPullingTime="1.93400981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:36:32.809957745 +0000 UTC m=+165.516033847" watchObservedRunningTime="2026-07-15 14:36:32.81149684 +0000 UTC m=+165.517572945" Jul 15 14:38:48.849389 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:38:48.849360 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 14:38:48.852605 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:38:48.852581 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 14:38:48.853430 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:38:48.853410 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 14:39:14.089006 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:14.088972 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zx982"] Jul 15 14:39:14.092153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:14.092138 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zx982" Jul 15 14:39:14.095220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:14.095200 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 15 14:39:14.095220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:14.095216 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 15 14:39:14.095393 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:14.095268 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-nq4h9\"" Jul 15 14:39:14.109348 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:14.109325 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zx982"] Jul 15 14:39:14.213223 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:14.213197 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z47ph\" (UniqueName: \"kubernetes.io/projected/42ba171d-f723-4e7f-bdfc-fdbeae6376dc-kube-api-access-z47ph\") pod \"cert-manager-operator-controller-manager-55c444bf86-zx982\" (UID: \"42ba171d-f723-4e7f-bdfc-fdbeae6376dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zx982" Jul 15 14:39:14.213324 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:14.213236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42ba171d-f723-4e7f-bdfc-fdbeae6376dc-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zx982\" (UID: \"42ba171d-f723-4e7f-bdfc-fdbeae6376dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zx982" Jul 15 14:39:14.314270 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:14.314244 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z47ph\" (UniqueName: \"kubernetes.io/projected/42ba171d-f723-4e7f-bdfc-fdbeae6376dc-kube-api-access-z47ph\") pod \"cert-manager-operator-controller-manager-55c444bf86-zx982\" (UID: \"42ba171d-f723-4e7f-bdfc-fdbeae6376dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zx982" Jul 15 14:39:14.314365 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:14.314283 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42ba171d-f723-4e7f-bdfc-fdbeae6376dc-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zx982\" (UID: \"42ba171d-f723-4e7f-bdfc-fdbeae6376dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zx982" Jul 15 14:39:14.314585 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:14.314571 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42ba171d-f723-4e7f-bdfc-fdbeae6376dc-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zx982\" (UID: \"42ba171d-f723-4e7f-bdfc-fdbeae6376dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zx982" Jul 15 14:39:14.327096 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:14.327070 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z47ph\" (UniqueName: \"kubernetes.io/projected/42ba171d-f723-4e7f-bdfc-fdbeae6376dc-kube-api-access-z47ph\") pod \"cert-manager-operator-controller-manager-55c444bf86-zx982\" (UID: \"42ba171d-f723-4e7f-bdfc-fdbeae6376dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zx982" Jul 15 14:39:14.401163 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:14.401104 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zx982" Jul 15 14:39:14.539016 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:14.538870 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zx982"] Jul 15 14:39:14.542080 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:39:14.542052 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ba171d_f723_4e7f_bdfc_fdbeae6376dc.slice/crio-fa5d6c058c2df105c3586502b5860125e90f44e1cf72cdcb158697676d5a1046 WatchSource:0}: Error finding container fa5d6c058c2df105c3586502b5860125e90f44e1cf72cdcb158697676d5a1046: Status 404 returned error can't find the container with id fa5d6c058c2df105c3586502b5860125e90f44e1cf72cdcb158697676d5a1046 Jul 15 14:39:14.545342 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:14.545328 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 14:39:15.354874 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:15.354825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zx982" event={"ID":"42ba171d-f723-4e7f-bdfc-fdbeae6376dc","Type":"ContainerStarted","Data":"fa5d6c058c2df105c3586502b5860125e90f44e1cf72cdcb158697676d5a1046"} Jul 15 14:39:17.365616 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:17.365581 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zx982" event={"ID":"42ba171d-f723-4e7f-bdfc-fdbeae6376dc","Type":"ContainerStarted","Data":"d7f409b41d70e4f91655c9115eb4cc8f19890080fef698f902d71074ef22550b"} Jul 15 14:39:17.393145 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:17.393083 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zx982" podStartSLOduration=1.077756386 podStartE2EDuration="3.393063376s" totalImagesPullingTime="2.31530699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:39:17.39006418 +0000 UTC m=+330.096140322" watchObservedRunningTime="2026-07-15 14:39:17.393063376 +0000 UTC m=+330.099139484" Jul 15 14:39:21.408189 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:21.408159 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-b4k74"] Jul 15 14:39:21.411385 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:21.411370 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-b4k74" Jul 15 14:39:21.420072 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:21.420047 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 14:39:21.421106 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:21.421092 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 14:39:21.421156 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:21.421103 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-k8rkx\"" Jul 15 14:39:21.437251 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:21.437225 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-b4k74"] Jul 15 14:39:21.570521 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:21.570489 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c3b351d-d109-42f6-977f-68bcaf004602-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-b4k74\" (UID: \"2c3b351d-d109-42f6-977f-68bcaf004602\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-b4k74" Jul 15 14:39:21.570635 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:21.570542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54zvr\" (UniqueName: \"kubernetes.io/projected/2c3b351d-d109-42f6-977f-68bcaf004602-kube-api-access-54zvr\") pod \"cert-manager-cainjector-559cdd8c87-b4k74\" (UID: \"2c3b351d-d109-42f6-977f-68bcaf004602\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-b4k74" Jul 15 14:39:21.670827 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:21.670774 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54zvr\" (UniqueName: \"kubernetes.io/projected/2c3b351d-d109-42f6-977f-68bcaf004602-kube-api-access-54zvr\") pod \"cert-manager-cainjector-559cdd8c87-b4k74\" (UID: \"2c3b351d-d109-42f6-977f-68bcaf004602\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-b4k74" Jul 15 14:39:21.670827 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:21.670819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c3b351d-d109-42f6-977f-68bcaf004602-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-b4k74\" (UID: \"2c3b351d-d109-42f6-977f-68bcaf004602\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-b4k74" Jul 15 14:39:21.688816 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:21.688791 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c3b351d-d109-42f6-977f-68bcaf004602-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-b4k74\" (UID: \"2c3b351d-d109-42f6-977f-68bcaf004602\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-b4k74" Jul 15 14:39:21.690220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:21.690201 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54zvr\" (UniqueName: \"kubernetes.io/projected/2c3b351d-d109-42f6-977f-68bcaf004602-kube-api-access-54zvr\") pod \"cert-manager-cainjector-559cdd8c87-b4k74\" (UID: \"2c3b351d-d109-42f6-977f-68bcaf004602\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-b4k74" Jul 15 14:39:21.719703 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:21.719682 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-b4k74" Jul 15 14:39:21.850429 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:21.850400 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-b4k74"] Jul 15 14:39:21.853621 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:39:21.853595 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c3b351d_d109_42f6_977f_68bcaf004602.slice/crio-da621b0b4234e9a60dc67c39286222990545d72121c80bbb96779984f48b6425 WatchSource:0}: Error finding container da621b0b4234e9a60dc67c39286222990545d72121c80bbb96779984f48b6425: Status 404 returned error can't find the container with id da621b0b4234e9a60dc67c39286222990545d72121c80bbb96779984f48b6425 Jul 15 14:39:22.385336 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:22.385303 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-b4k74" event={"ID":"2c3b351d-d109-42f6-977f-68bcaf004602","Type":"ContainerStarted","Data":"da621b0b4234e9a60dc67c39286222990545d72121c80bbb96779984f48b6425"} Jul 15 14:39:26.401065 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:26.401025 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-b4k74" event={"ID":"2c3b351d-d109-42f6-977f-68bcaf004602","Type":"ContainerStarted","Data":"910ebc7e8010d46b37a098c34f05b8330a17db5baf1687dd0d743cd3574f2e38"} Jul 15 14:39:38.454313 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:38.454252 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-b4k74" podStartSLOduration=13.956945308 podStartE2EDuration="17.454232475s" totalImagesPullingTime="3.497287167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:39:26.44369133 +0000 UTC m=+339.149767434" watchObservedRunningTime="2026-07-15 14:39:38.454232475 +0000 UTC m=+351.160308618" Jul 15 14:39:38.454786 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:38.454771 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sbthk"] Jul 15 14:39:38.458066 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:38.458047 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-sbthk" Jul 15 14:39:38.461109 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:38.461089 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ztjzx\"" Jul 15 14:39:38.468263 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:38.468240 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sbthk"] Jul 15 14:39:38.502739 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:38.502699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/208c1aa3-e4ab-4d9e-badd-2f0a13c4e3ff-bound-sa-token\") pod \"cert-manager-55b4d954d5-sbthk\" (UID: \"208c1aa3-e4ab-4d9e-badd-2f0a13c4e3ff\") " pod="cert-manager/cert-manager-55b4d954d5-sbthk" Jul 15 14:39:38.502952 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:38.502761 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kngw7\" (UniqueName: \"kubernetes.io/projected/208c1aa3-e4ab-4d9e-badd-2f0a13c4e3ff-kube-api-access-kngw7\") pod \"cert-manager-55b4d954d5-sbthk\" (UID: \"208c1aa3-e4ab-4d9e-badd-2f0a13c4e3ff\") " pod="cert-manager/cert-manager-55b4d954d5-sbthk" Jul 15 14:39:38.603225 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:38.603180 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/208c1aa3-e4ab-4d9e-badd-2f0a13c4e3ff-bound-sa-token\") pod \"cert-manager-55b4d954d5-sbthk\" (UID: \"208c1aa3-e4ab-4d9e-badd-2f0a13c4e3ff\") " pod="cert-manager/cert-manager-55b4d954d5-sbthk" Jul 15 14:39:38.603410 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:38.603244 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kngw7\" (UniqueName: \"kubernetes.io/projected/208c1aa3-e4ab-4d9e-badd-2f0a13c4e3ff-kube-api-access-kngw7\") pod \"cert-manager-55b4d954d5-sbthk\" (UID: \"208c1aa3-e4ab-4d9e-badd-2f0a13c4e3ff\") " pod="cert-manager/cert-manager-55b4d954d5-sbthk" Jul 15 14:39:38.614090 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:38.614057 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/208c1aa3-e4ab-4d9e-badd-2f0a13c4e3ff-bound-sa-token\") pod \"cert-manager-55b4d954d5-sbthk\" (UID: \"208c1aa3-e4ab-4d9e-badd-2f0a13c4e3ff\") " pod="cert-manager/cert-manager-55b4d954d5-sbthk" Jul 15 14:39:38.614267 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:38.614124 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kngw7\" (UniqueName: \"kubernetes.io/projected/208c1aa3-e4ab-4d9e-badd-2f0a13c4e3ff-kube-api-access-kngw7\") pod \"cert-manager-55b4d954d5-sbthk\" (UID: \"208c1aa3-e4ab-4d9e-badd-2f0a13c4e3ff\") " pod="cert-manager/cert-manager-55b4d954d5-sbthk" Jul 15 14:39:38.766784 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:38.766686 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-sbthk" Jul 15 14:39:38.898022 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:38.897992 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sbthk"] Jul 15 14:39:38.901281 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:39:38.901253 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod208c1aa3_e4ab_4d9e_badd_2f0a13c4e3ff.slice/crio-64349722198d0c916ff4f20b368c63993ee02264e021011ca606a679d97e80ad WatchSource:0}: Error finding container 64349722198d0c916ff4f20b368c63993ee02264e021011ca606a679d97e80ad: Status 404 returned error can't find the container with id 64349722198d0c916ff4f20b368c63993ee02264e021011ca606a679d97e80ad Jul 15 14:39:39.454174 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:39.454095 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-sbthk" event={"ID":"208c1aa3-e4ab-4d9e-badd-2f0a13c4e3ff","Type":"ContainerStarted","Data":"07e868db6c3d209696a00d9670818b61813066d5b0b0162929a810cf5df49736"} Jul 15 14:39:39.454174 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:39.454138 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-sbthk" event={"ID":"208c1aa3-e4ab-4d9e-badd-2f0a13c4e3ff","Type":"ContainerStarted","Data":"64349722198d0c916ff4f20b368c63993ee02264e021011ca606a679d97e80ad"} Jul 15 14:39:39.480371 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:39.480328 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-sbthk" podStartSLOduration=1.480313244 podStartE2EDuration="1.480313244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:39:39.47891864 +0000 UTC m=+352.184994744" watchObservedRunningTime="2026-07-15 14:39:39.480313244 +0000 UTC m=+352.186389348" Jul 15 14:39:54.257658 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.257624 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr"] Jul 15 14:39:54.262181 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.262158 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.266223 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.266199 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 14:39:54.266403 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.266376 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 14:39:54.266403 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.266381 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vdrdb\"" Jul 15 14:39:54.266551 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.266431 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 14:39:54.270717 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.270698 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 14:39:54.310903 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.310866 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr"] Jul 15 14:39:54.334697 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.334676 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90bc98c4-0476-47c2-a934-24ec14e2be01-tmp\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr\" (UID: \"90bc98c4-0476-47c2-a934-24ec14e2be01\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.334786 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.334717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90bc98c4-0476-47c2-a934-24ec14e2be01-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr\" (UID: \"90bc98c4-0476-47c2-a934-24ec14e2be01\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.334786 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.334751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90bc98c4-0476-47c2-a934-24ec14e2be01-manifests\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr\" (UID: \"90bc98c4-0476-47c2-a934-24ec14e2be01\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.334866 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.334790 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/90bc98c4-0476-47c2-a934-24ec14e2be01-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr\" (UID: \"90bc98c4-0476-47c2-a934-24ec14e2be01\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.334866 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.334809 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx878\" (UniqueName: \"kubernetes.io/projected/90bc98c4-0476-47c2-a934-24ec14e2be01-kube-api-access-qx878\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr\" (UID: \"90bc98c4-0476-47c2-a934-24ec14e2be01\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.435605 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.435580 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90bc98c4-0476-47c2-a934-24ec14e2be01-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr\" (UID: \"90bc98c4-0476-47c2-a934-24ec14e2be01\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.435703 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.435612 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90bc98c4-0476-47c2-a934-24ec14e2be01-manifests\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr\" (UID: \"90bc98c4-0476-47c2-a934-24ec14e2be01\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.435703 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.435649 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/90bc98c4-0476-47c2-a934-24ec14e2be01-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr\" (UID: \"90bc98c4-0476-47c2-a934-24ec14e2be01\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.435703 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.435670 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx878\" (UniqueName: \"kubernetes.io/projected/90bc98c4-0476-47c2-a934-24ec14e2be01-kube-api-access-qx878\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr\" (UID: \"90bc98c4-0476-47c2-a934-24ec14e2be01\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.435703 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.435686 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90bc98c4-0476-47c2-a934-24ec14e2be01-tmp\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr\" (UID: \"90bc98c4-0476-47c2-a934-24ec14e2be01\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.436079 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.436053 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90bc98c4-0476-47c2-a934-24ec14e2be01-manifests\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr\" (UID: \"90bc98c4-0476-47c2-a934-24ec14e2be01\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.438034 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.438012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90bc98c4-0476-47c2-a934-24ec14e2be01-tmp\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr\" (UID: \"90bc98c4-0476-47c2-a934-24ec14e2be01\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.438110 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.438033 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/90bc98c4-0476-47c2-a934-24ec14e2be01-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr\" (UID: \"90bc98c4-0476-47c2-a934-24ec14e2be01\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.438157 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.438143 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90bc98c4-0476-47c2-a934-24ec14e2be01-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr\" (UID: \"90bc98c4-0476-47c2-a934-24ec14e2be01\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.454278 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.454255 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx878\" (UniqueName: \"kubernetes.io/projected/90bc98c4-0476-47c2-a934-24ec14e2be01-kube-api-access-qx878\") pod \"opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr\" (UID: \"90bc98c4-0476-47c2-a934-24ec14e2be01\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.572102 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.572074 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:39:54.703839 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:54.703810 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr"] Jul 15 14:39:54.707075 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:39:54.707051 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bc98c4_0476_47c2_a934_24ec14e2be01.slice/crio-1dce439f5dd463cc375d23e5fd9b0ab0cbd1c0980676a00adaa78f0482c53fce WatchSource:0}: Error finding container 1dce439f5dd463cc375d23e5fd9b0ab0cbd1c0980676a00adaa78f0482c53fce: Status 404 returned error can't find the container with id 1dce439f5dd463cc375d23e5fd9b0ab0cbd1c0980676a00adaa78f0482c53fce Jul 15 14:39:55.522469 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:55.522427 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" event={"ID":"90bc98c4-0476-47c2-a934-24ec14e2be01","Type":"ContainerStarted","Data":"1dce439f5dd463cc375d23e5fd9b0ab0cbd1c0980676a00adaa78f0482c53fce"} Jul 15 14:39:58.537175 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:58.537139 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" event={"ID":"90bc98c4-0476-47c2-a934-24ec14e2be01","Type":"ContainerStarted","Data":"208ecd07688a6cffc1d578df9d075e927fd93705ffd59d56e0805d81681f96a2"} Jul 15 14:39:59.542792 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:59.542758 2574 generic.go:356] "Generic (PLEG): container finished" podID="90bc98c4-0476-47c2-a934-24ec14e2be01" containerID="208ecd07688a6cffc1d578df9d075e927fd93705ffd59d56e0805d81681f96a2" exitCode=0 Jul 15 14:39:59.543230 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:39:59.542837 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" event={"ID":"90bc98c4-0476-47c2-a934-24ec14e2be01","Type":"ContainerDied","Data":"208ecd07688a6cffc1d578df9d075e927fd93705ffd59d56e0805d81681f96a2"} Jul 15 14:40:00.548800 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:00.548767 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" event={"ID":"90bc98c4-0476-47c2-a934-24ec14e2be01","Type":"ContainerStarted","Data":"5d9fb8b51fcc789e5c7c02bfd10724441c50831448db5e1fd4cc181776379c90"} Jul 15 14:40:00.549208 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:00.548861 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:40:00.649681 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:00.649635 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" podStartSLOduration=2.090199411 podStartE2EDuration="6.649619995s" totalImagesPullingTime="3.467779877s" totalInitContainerRuntime="1.091640707s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:39:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:40:00.648416841 +0000 UTC m=+373.354492947" watchObservedRunningTime="2026-07-15 14:40:00.649619995 +0000 UTC m=+373.355696141" Jul 15 14:40:11.213357 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.213327 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68975544d6-jldvj"] Jul 15 14:40:11.216673 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.216650 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:11.219143 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.219123 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 14:40:11.220329 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.220311 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 14:40:11.220329 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.220324 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 14:40:11.220465 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.220313 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 14:40:11.220734 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.220722 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mz6sr\"" Jul 15 14:40:11.224204 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.224189 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 14:40:11.228877 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.228859 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68975544d6-jldvj"] Jul 15 14:40:11.366701 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.366670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsc98\" (UniqueName: \"kubernetes.io/projected/efe244d8-1cf0-488d-ac8b-ae6847f2f1bc-kube-api-access-wsc98\") pod \"lws-controller-manager-68975544d6-jldvj\" (UID: \"efe244d8-1cf0-488d-ac8b-ae6847f2f1bc\") " pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:11.366822 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.366711 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/efe244d8-1cf0-488d-ac8b-ae6847f2f1bc-metrics-cert\") pod \"lws-controller-manager-68975544d6-jldvj\" (UID: \"efe244d8-1cf0-488d-ac8b-ae6847f2f1bc\") " pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:11.366822 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.366731 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/efe244d8-1cf0-488d-ac8b-ae6847f2f1bc-manager-config\") pod \"lws-controller-manager-68975544d6-jldvj\" (UID: \"efe244d8-1cf0-488d-ac8b-ae6847f2f1bc\") " pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:11.366959 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.366814 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efe244d8-1cf0-488d-ac8b-ae6847f2f1bc-cert\") pod \"lws-controller-manager-68975544d6-jldvj\" (UID: \"efe244d8-1cf0-488d-ac8b-ae6847f2f1bc\") " pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:11.467975 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.467907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsc98\" (UniqueName: \"kubernetes.io/projected/efe244d8-1cf0-488d-ac8b-ae6847f2f1bc-kube-api-access-wsc98\") pod \"lws-controller-manager-68975544d6-jldvj\" (UID: \"efe244d8-1cf0-488d-ac8b-ae6847f2f1bc\") " pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:11.467975 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.467958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/efe244d8-1cf0-488d-ac8b-ae6847f2f1bc-metrics-cert\") pod \"lws-controller-manager-68975544d6-jldvj\" (UID: \"efe244d8-1cf0-488d-ac8b-ae6847f2f1bc\") " pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:11.468134 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.467987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/efe244d8-1cf0-488d-ac8b-ae6847f2f1bc-manager-config\") pod \"lws-controller-manager-68975544d6-jldvj\" (UID: \"efe244d8-1cf0-488d-ac8b-ae6847f2f1bc\") " pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:11.468134 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.468018 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efe244d8-1cf0-488d-ac8b-ae6847f2f1bc-cert\") pod \"lws-controller-manager-68975544d6-jldvj\" (UID: \"efe244d8-1cf0-488d-ac8b-ae6847f2f1bc\") " pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:11.468597 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.468575 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/efe244d8-1cf0-488d-ac8b-ae6847f2f1bc-manager-config\") pod \"lws-controller-manager-68975544d6-jldvj\" (UID: \"efe244d8-1cf0-488d-ac8b-ae6847f2f1bc\") " pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:11.470299 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.470278 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efe244d8-1cf0-488d-ac8b-ae6847f2f1bc-cert\") pod \"lws-controller-manager-68975544d6-jldvj\" (UID: \"efe244d8-1cf0-488d-ac8b-ae6847f2f1bc\") " pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:11.470378 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.470362 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/efe244d8-1cf0-488d-ac8b-ae6847f2f1bc-metrics-cert\") pod \"lws-controller-manager-68975544d6-jldvj\" (UID: \"efe244d8-1cf0-488d-ac8b-ae6847f2f1bc\") " pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:11.480008 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.479981 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsc98\" (UniqueName: \"kubernetes.io/projected/efe244d8-1cf0-488d-ac8b-ae6847f2f1bc-kube-api-access-wsc98\") pod \"lws-controller-manager-68975544d6-jldvj\" (UID: \"efe244d8-1cf0-488d-ac8b-ae6847f2f1bc\") " pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:11.525447 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.525421 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:11.555004 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.554985 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr" Jul 15 14:40:11.675335 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:11.675292 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68975544d6-jldvj"] Jul 15 14:40:11.677836 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:40:11.677813 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe244d8_1cf0_488d_ac8b_ae6847f2f1bc.slice/crio-8c140de46523d8ec5354931af3a371e11de1880017e62f96725c13e38b3ab80a WatchSource:0}: Error finding container 8c140de46523d8ec5354931af3a371e11de1880017e62f96725c13e38b3ab80a: Status 404 returned error can't find the container with id 8c140de46523d8ec5354931af3a371e11de1880017e62f96725c13e38b3ab80a Jul 15 14:40:12.600648 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:12.600615 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" event={"ID":"efe244d8-1cf0-488d-ac8b-ae6847f2f1bc","Type":"ContainerStarted","Data":"8c140de46523d8ec5354931af3a371e11de1880017e62f96725c13e38b3ab80a"} Jul 15 14:40:15.615874 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:15.615810 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" event={"ID":"efe244d8-1cf0-488d-ac8b-ae6847f2f1bc","Type":"ContainerStarted","Data":"409c32b83003aaf10522734eb7c17ab54391b98c2255565462bce3e80881fd78"} Jul 15 14:40:15.616324 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:15.616031 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:15.646564 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:15.646517 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" podStartSLOduration=0.870673044 podStartE2EDuration="4.646498401s" totalImagesPullingTime="3.775825357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:40:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:40:15.646169674 +0000 UTC m=+388.352245778" watchObservedRunningTime="2026-07-15 14:40:15.646498401 +0000 UTC m=+388.352574504" Jul 15 14:40:26.623340 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:26.623308 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68975544d6-jldvj" Jul 15 14:40:35.983370 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:35.983334 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h"] Jul 15 14:40:35.990590 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:35.990572 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:35.994455 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:35.994432 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 14:40:35.994601 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:35.994586 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4khz4\"" Jul 15 14:40:35.994801 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:35.994786 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 14:40:35.995384 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:35.995368 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 14:40:36.010530 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.010504 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h"] Jul 15 14:40:36.056414 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.056389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f2caa4-a7a4-483d-98d1-03f471c890e9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.056502 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.056416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f2caa4-a7a4-483d-98d1-03f471c890e9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.056502 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.056437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f2caa4-a7a4-483d-98d1-03f471c890e9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.056502 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.056457 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f2caa4-a7a4-483d-98d1-03f471c890e9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.056502 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.056493 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2caa4-a7a4-483d-98d1-03f471c890e9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.056625 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.056519 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f2caa4-a7a4-483d-98d1-03f471c890e9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.056625 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.056544 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2caa4-a7a4-483d-98d1-03f471c890e9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.056625 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.056590 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f2caa4-a7a4-483d-98d1-03f471c890e9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.056625 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.056619 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmrrn\" (UniqueName: \"kubernetes.io/projected/36f2caa4-a7a4-483d-98d1-03f471c890e9-kube-api-access-mmrrn\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.157104 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.157080 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmrrn\" (UniqueName: \"kubernetes.io/projected/36f2caa4-a7a4-483d-98d1-03f471c890e9-kube-api-access-mmrrn\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.157189 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.157120 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f2caa4-a7a4-483d-98d1-03f471c890e9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.157189 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.157140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f2caa4-a7a4-483d-98d1-03f471c890e9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.157189 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.157160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f2caa4-a7a4-483d-98d1-03f471c890e9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.157315 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.157192 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f2caa4-a7a4-483d-98d1-03f471c890e9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.157315 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.157219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2caa4-a7a4-483d-98d1-03f471c890e9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.157315 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.157246 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f2caa4-a7a4-483d-98d1-03f471c890e9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.157315 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.157275 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2caa4-a7a4-483d-98d1-03f471c890e9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.157315 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.157310 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f2caa4-a7a4-483d-98d1-03f471c890e9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.157672 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.157649 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2caa4-a7a4-483d-98d1-03f471c890e9-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.157743 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.157704 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36f2caa4-a7a4-483d-98d1-03f471c890e9-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.157875 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.157844 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36f2caa4-a7a4-483d-98d1-03f471c890e9-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.157875 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.157860 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36f2caa4-a7a4-483d-98d1-03f471c890e9-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.157995 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.157912 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36f2caa4-a7a4-483d-98d1-03f471c890e9-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.159523 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.159504 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36f2caa4-a7a4-483d-98d1-03f471c890e9-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.159678 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.159661 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36f2caa4-a7a4-483d-98d1-03f471c890e9-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.169041 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.169021 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmrrn\" (UniqueName: \"kubernetes.io/projected/36f2caa4-a7a4-483d-98d1-03f471c890e9-kube-api-access-mmrrn\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.169178 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.169161 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36f2caa4-a7a4-483d-98d1-03f471c890e9-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mm47h\" (UID: \"36f2caa4-a7a4-483d-98d1-03f471c890e9\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.299647 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.299577 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:36.436020 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.435999 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h"] Jul 15 14:40:36.438107 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:40:36.438080 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f2caa4_a7a4_483d_98d1_03f471c890e9.slice/crio-27d5506be795147f5d125c11dcaaa96db63d2ae1fdf36f264ded5844168e1016 WatchSource:0}: Error finding container 27d5506be795147f5d125c11dcaaa96db63d2ae1fdf36f264ded5844168e1016: Status 404 returned error can't find the container with id 27d5506be795147f5d125c11dcaaa96db63d2ae1fdf36f264ded5844168e1016 Jul 15 14:40:36.703130 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:36.703100 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" event={"ID":"36f2caa4-a7a4-483d-98d1-03f471c890e9","Type":"ContainerStarted","Data":"27d5506be795147f5d125c11dcaaa96db63d2ae1fdf36f264ded5844168e1016"} Jul 15 14:40:38.692811 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:38.692772 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:40:38.693197 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:38.692863 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:40:38.693197 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:38.692928 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:40:39.717381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:39.717350 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" event={"ID":"36f2caa4-a7a4-483d-98d1-03f471c890e9","Type":"ContainerStarted","Data":"fdb1be25d41281601342888f78d5c4afd071c22c268f4ccec71fbb90dc1d86f1"} Jul 15 14:40:39.742998 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:39.742940 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" podStartSLOduration=2.490405919 podStartE2EDuration="4.742920534s" totalImagesPullingTime="2.252514615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:40:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:40:39.741239666 +0000 UTC m=+412.447315770" watchObservedRunningTime="2026-07-15 14:40:39.742920534 +0000 UTC m=+412.448996638" Jul 15 14:40:40.300680 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:40.300643 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:40.301830 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:40.301807 2574 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6576859cf4mm47h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Jul 15 14:40:40.301950 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:40.301866 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" podUID="36f2caa4-a7a4-483d-98d1-03f471c890e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Jul 15 14:40:41.303399 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:41.303367 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:41.725552 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:41.725470 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:40:41.726550 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:40:41.726533 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mm47h" Jul 15 14:41:17.387821 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.387749 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kct7c"] Jul 15 14:41:17.389879 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.389861 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kct7c" Jul 15 14:41:17.394329 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.394262 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-4r2gv\"" Jul 15 14:41:17.395287 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.395263 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 14:41:17.395477 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.395357 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 14:41:17.402623 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.402575 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kct7c"] Jul 15 14:41:17.492695 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.492666 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdptn\" (UniqueName: \"kubernetes.io/projected/fa8b4634-8681-4cee-9ecc-ddffee8b7613-kube-api-access-pdptn\") pod \"kuadrant-operator-catalog-kct7c\" (UID: \"fa8b4634-8681-4cee-9ecc-ddffee8b7613\") " pod="kuadrant-system/kuadrant-operator-catalog-kct7c" Jul 15 14:41:17.593533 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.593505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdptn\" (UniqueName: \"kubernetes.io/projected/fa8b4634-8681-4cee-9ecc-ddffee8b7613-kube-api-access-pdptn\") pod \"kuadrant-operator-catalog-kct7c\" (UID: \"fa8b4634-8681-4cee-9ecc-ddffee8b7613\") " pod="kuadrant-system/kuadrant-operator-catalog-kct7c" Jul 15 14:41:17.604706 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.604682 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdptn\" (UniqueName: \"kubernetes.io/projected/fa8b4634-8681-4cee-9ecc-ddffee8b7613-kube-api-access-pdptn\") pod \"kuadrant-operator-catalog-kct7c\" (UID: \"fa8b4634-8681-4cee-9ecc-ddffee8b7613\") " pod="kuadrant-system/kuadrant-operator-catalog-kct7c" Jul 15 14:41:17.699463 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.699408 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kct7c"] Jul 15 14:41:17.699546 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.699477 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kct7c" Jul 15 14:41:17.854553 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.854529 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kct7c"] Jul 15 14:41:17.857108 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:41:17.857081 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8b4634_8681_4cee_9ecc_ddffee8b7613.slice/crio-35e051b38601b4809eb59d743c42da1cb43168c69707d82a1f5d08526f02e67d WatchSource:0}: Error finding container 35e051b38601b4809eb59d743c42da1cb43168c69707d82a1f5d08526f02e67d: Status 404 returned error can't find the container with id 35e051b38601b4809eb59d743c42da1cb43168c69707d82a1f5d08526f02e67d Jul 15 14:41:17.882827 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.882797 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kct7c" event={"ID":"fa8b4634-8681-4cee-9ecc-ddffee8b7613","Type":"ContainerStarted","Data":"35e051b38601b4809eb59d743c42da1cb43168c69707d82a1f5d08526f02e67d"} Jul 15 14:41:17.942315 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.942285 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s5b7f"] Jul 15 14:41:17.945024 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.945008 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-s5b7f" Jul 15 14:41:17.959112 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:17.959089 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s5b7f"] Jul 15 14:41:18.098030 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:18.097998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kqgx\" (UniqueName: \"kubernetes.io/projected/a657d514-40bf-4944-b6b3-3dc8a2293643-kube-api-access-9kqgx\") pod \"kuadrant-operator-catalog-s5b7f\" (UID: \"a657d514-40bf-4944-b6b3-3dc8a2293643\") " pod="kuadrant-system/kuadrant-operator-catalog-s5b7f" Jul 15 14:41:18.198735 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:18.198703 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kqgx\" (UniqueName: \"kubernetes.io/projected/a657d514-40bf-4944-b6b3-3dc8a2293643-kube-api-access-9kqgx\") pod \"kuadrant-operator-catalog-s5b7f\" (UID: \"a657d514-40bf-4944-b6b3-3dc8a2293643\") " pod="kuadrant-system/kuadrant-operator-catalog-s5b7f" Jul 15 14:41:18.212178 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:18.212117 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kqgx\" (UniqueName: \"kubernetes.io/projected/a657d514-40bf-4944-b6b3-3dc8a2293643-kube-api-access-9kqgx\") pod \"kuadrant-operator-catalog-s5b7f\" (UID: \"a657d514-40bf-4944-b6b3-3dc8a2293643\") " pod="kuadrant-system/kuadrant-operator-catalog-s5b7f" Jul 15 14:41:18.254317 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:18.254286 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-s5b7f" Jul 15 14:41:18.397553 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:18.397519 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-s5b7f"] Jul 15 14:41:18.405603 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:41:18.405580 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda657d514_40bf_4944_b6b3_3dc8a2293643.slice/crio-50d26570d1133ee039df5a76dad896a4c83036540ed5f484f4b9898de3dad304 WatchSource:0}: Error finding container 50d26570d1133ee039df5a76dad896a4c83036540ed5f484f4b9898de3dad304: Status 404 returned error can't find the container with id 50d26570d1133ee039df5a76dad896a4c83036540ed5f484f4b9898de3dad304 Jul 15 14:41:18.888355 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:18.888318 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-s5b7f" event={"ID":"a657d514-40bf-4944-b6b3-3dc8a2293643","Type":"ContainerStarted","Data":"50d26570d1133ee039df5a76dad896a4c83036540ed5f484f4b9898de3dad304"} Jul 15 14:41:21.903300 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:21.903262 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kct7c" event={"ID":"fa8b4634-8681-4cee-9ecc-ddffee8b7613","Type":"ContainerStarted","Data":"f1cc656543fd316488c8a3c880fc9b9942be6479bd4412a4de5b7a88d42f94bd"} Jul 15 14:41:21.903732 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:21.903346 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-kct7c" podUID="fa8b4634-8681-4cee-9ecc-ddffee8b7613" containerName="registry-server" containerID="cri-o://f1cc656543fd316488c8a3c880fc9b9942be6479bd4412a4de5b7a88d42f94bd" gracePeriod=2 Jul 15 14:41:21.926798 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:21.926754 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-kct7c" podStartSLOduration=1.527954295 podStartE2EDuration="4.926740044s" totalImagesPullingTime="3.398785749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:41:21.92518862 +0000 UTC m=+454.631264724" watchObservedRunningTime="2026-07-15 14:41:21.926740044 +0000 UTC m=+454.632816147" Jul 15 14:41:22.153093 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:22.153070 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kct7c" Jul 15 14:41:22.336259 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:22.336220 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdptn\" (UniqueName: \"kubernetes.io/projected/fa8b4634-8681-4cee-9ecc-ddffee8b7613-kube-api-access-pdptn\") pod \"fa8b4634-8681-4cee-9ecc-ddffee8b7613\" (UID: \"fa8b4634-8681-4cee-9ecc-ddffee8b7613\") " Jul 15 14:41:22.338238 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:22.338210 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8b4634-8681-4cee-9ecc-ddffee8b7613-kube-api-access-pdptn" (OuterVolumeSpecName: "kube-api-access-pdptn") pod "fa8b4634-8681-4cee-9ecc-ddffee8b7613" (UID: "fa8b4634-8681-4cee-9ecc-ddffee8b7613"). InnerVolumeSpecName "kube-api-access-pdptn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:41:22.437491 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:22.437459 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdptn\" (UniqueName: \"kubernetes.io/projected/fa8b4634-8681-4cee-9ecc-ddffee8b7613-kube-api-access-pdptn\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:41:22.908707 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:22.908677 2574 generic.go:356] "Generic (PLEG): container finished" podID="fa8b4634-8681-4cee-9ecc-ddffee8b7613" containerID="f1cc656543fd316488c8a3c880fc9b9942be6479bd4412a4de5b7a88d42f94bd" exitCode=0 Jul 15 14:41:22.909222 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:22.908714 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kct7c" Jul 15 14:41:22.909222 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:22.908745 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kct7c" event={"ID":"fa8b4634-8681-4cee-9ecc-ddffee8b7613","Type":"ContainerDied","Data":"f1cc656543fd316488c8a3c880fc9b9942be6479bd4412a4de5b7a88d42f94bd"} Jul 15 14:41:22.909222 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:22.908791 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kct7c" event={"ID":"fa8b4634-8681-4cee-9ecc-ddffee8b7613","Type":"ContainerDied","Data":"35e051b38601b4809eb59d743c42da1cb43168c69707d82a1f5d08526f02e67d"} Jul 15 14:41:22.909222 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:22.908820 2574 scope.go:117] "RemoveContainer" containerID="f1cc656543fd316488c8a3c880fc9b9942be6479bd4412a4de5b7a88d42f94bd" Jul 15 14:41:22.910253 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:22.910231 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-s5b7f" event={"ID":"a657d514-40bf-4944-b6b3-3dc8a2293643","Type":"ContainerStarted","Data":"dd67cb2065f20487258a5805ce5eeab08f20b770b65823db4dcc64c3c3e0e041"} Jul 15 14:41:22.917302 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:22.917288 2574 scope.go:117] "RemoveContainer" containerID="f1cc656543fd316488c8a3c880fc9b9942be6479bd4412a4de5b7a88d42f94bd" Jul 15 14:41:22.917536 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:41:22.917521 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1cc656543fd316488c8a3c880fc9b9942be6479bd4412a4de5b7a88d42f94bd\": container with ID starting with f1cc656543fd316488c8a3c880fc9b9942be6479bd4412a4de5b7a88d42f94bd not found: ID does not exist" containerID="f1cc656543fd316488c8a3c880fc9b9942be6479bd4412a4de5b7a88d42f94bd" Jul 15 14:41:22.917583 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:22.917542 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cc656543fd316488c8a3c880fc9b9942be6479bd4412a4de5b7a88d42f94bd"} err="failed to get container status \"f1cc656543fd316488c8a3c880fc9b9942be6479bd4412a4de5b7a88d42f94bd\": rpc error: code = NotFound desc = could not find container \"f1cc656543fd316488c8a3c880fc9b9942be6479bd4412a4de5b7a88d42f94bd\": container with ID starting with f1cc656543fd316488c8a3c880fc9b9942be6479bd4412a4de5b7a88d42f94bd not found: ID does not exist" Jul 15 14:41:22.935110 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:22.935072 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-s5b7f" podStartSLOduration=2.123561592 podStartE2EDuration="5.93506072s" totalImagesPullingTime="3.811499128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:41:22.933318824 +0000 UTC m=+455.639394947" watchObservedRunningTime="2026-07-15 14:41:22.93506072 +0000 UTC m=+455.641136823" Jul 15 14:41:22.948948 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:22.948925 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kct7c"] Jul 15 14:41:22.953648 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:22.953630 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kct7c"] Jul 15 14:41:23.082647 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:23.082617 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa8b4634-8681-4cee-9ecc-ddffee8b7613" path="/var/lib/kubelet/pods/fa8b4634-8681-4cee-9ecc-ddffee8b7613/volumes" Jul 15 14:41:28.255220 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:28.255186 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-s5b7f" Jul 15 14:41:28.255599 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:28.255245 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-s5b7f" Jul 15 14:41:28.276651 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:28.276628 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-s5b7f" Jul 15 14:41:28.958449 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:28.958413 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-s5b7f" Jul 15 14:41:50.142208 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:50.142175 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq"] Jul 15 14:41:50.142541 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:50.142459 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa8b4634-8681-4cee-9ecc-ddffee8b7613" containerName="registry-server" Jul 15 14:41:50.142541 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:50.142469 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa8b4634-8681-4cee-9ecc-ddffee8b7613" containerName="registry-server" Jul 15 14:41:50.142541 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:50.142535 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa8b4634-8681-4cee-9ecc-ddffee8b7613" containerName="registry-server" Jul 15 14:41:50.146548 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:50.146531 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" Jul 15 14:41:50.150791 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:50.150773 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-cs5t8\"" Jul 15 14:41:50.173750 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:50.173725 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq"] Jul 15 14:41:50.255453 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:50.255428 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d69mx\" (UniqueName: \"kubernetes.io/projected/8600327f-7641-4e0c-b86a-69bb81aad3ca-kube-api-access-d69mx\") pod \"limitador-operator-controller-manager-859d7b78bd-wxxhq\" (UID: \"8600327f-7641-4e0c-b86a-69bb81aad3ca\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" Jul 15 14:41:50.356259 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:50.356228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d69mx\" (UniqueName: \"kubernetes.io/projected/8600327f-7641-4e0c-b86a-69bb81aad3ca-kube-api-access-d69mx\") pod \"limitador-operator-controller-manager-859d7b78bd-wxxhq\" (UID: \"8600327f-7641-4e0c-b86a-69bb81aad3ca\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" Jul 15 14:41:50.401484 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:50.401433 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d69mx\" (UniqueName: \"kubernetes.io/projected/8600327f-7641-4e0c-b86a-69bb81aad3ca-kube-api-access-d69mx\") pod \"limitador-operator-controller-manager-859d7b78bd-wxxhq\" (UID: \"8600327f-7641-4e0c-b86a-69bb81aad3ca\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" Jul 15 14:41:50.456415 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:50.456395 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" Jul 15 14:41:50.594949 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:41:50.594922 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8600327f_7641_4e0c_b86a_69bb81aad3ca.slice/crio-b95c1e119eeaed678411a53fbef7c3383dbf9dd4e09d0ae8e57e8772ad11878e WatchSource:0}: Error finding container b95c1e119eeaed678411a53fbef7c3383dbf9dd4e09d0ae8e57e8772ad11878e: Status 404 returned error can't find the container with id b95c1e119eeaed678411a53fbef7c3383dbf9dd4e09d0ae8e57e8772ad11878e Jul 15 14:41:50.595297 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:50.595272 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq"] Jul 15 14:41:51.030719 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:51.030682 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" event={"ID":"8600327f-7641-4e0c-b86a-69bb81aad3ca","Type":"ContainerStarted","Data":"b95c1e119eeaed678411a53fbef7c3383dbf9dd4e09d0ae8e57e8772ad11878e"} Jul 15 14:41:53.040754 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:53.040717 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" event={"ID":"8600327f-7641-4e0c-b86a-69bb81aad3ca","Type":"ContainerStarted","Data":"9f1b0daf102d4f278979897c5ad16aa0a8ee24717d72ea68b8497a6d076e8a61"} Jul 15 14:41:53.041146 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:41:53.040825 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" Jul 15 14:42:03.727572 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:03.727522 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" podStartSLOduration=11.8551762 podStartE2EDuration="13.727506465s" totalImagesPullingTime="1.872330265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:41:53.101970379 +0000 UTC m=+485.808046483" watchObservedRunningTime="2026-07-15 14:42:03.727506465 +0000 UTC m=+496.433582570" Jul 15 14:42:03.728032 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:03.727938 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ts4lq"] Jul 15 14:42:03.730259 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:03.730243 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ts4lq" Jul 15 14:42:03.735567 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:03.735543 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-vfc6n\"" Jul 15 14:42:03.749862 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:03.749839 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ts4lq"] Jul 15 14:42:03.868557 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:03.868525 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn2sq\" (UniqueName: \"kubernetes.io/projected/e5a03076-1c4b-4cde-b963-862cdb07c9cd-kube-api-access-rn2sq\") pod \"authorino-operator-75cc8d5cb7-ts4lq\" (UID: \"e5a03076-1c4b-4cde-b963-862cdb07c9cd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ts4lq" Jul 15 14:42:03.969726 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:03.969697 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn2sq\" (UniqueName: \"kubernetes.io/projected/e5a03076-1c4b-4cde-b963-862cdb07c9cd-kube-api-access-rn2sq\") pod \"authorino-operator-75cc8d5cb7-ts4lq\" (UID: \"e5a03076-1c4b-4cde-b963-862cdb07c9cd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ts4lq" Jul 15 14:42:03.983786 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:03.983719 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn2sq\" (UniqueName: \"kubernetes.io/projected/e5a03076-1c4b-4cde-b963-862cdb07c9cd-kube-api-access-rn2sq\") pod \"authorino-operator-75cc8d5cb7-ts4lq\" (UID: \"e5a03076-1c4b-4cde-b963-862cdb07c9cd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ts4lq" Jul 15 14:42:04.040211 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:04.040185 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ts4lq" Jul 15 14:42:04.047894 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:04.047862 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" Jul 15 14:42:04.171165 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:04.171138 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ts4lq"] Jul 15 14:42:04.173781 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:42:04.173750 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a03076_1c4b_4cde_b963_862cdb07c9cd.slice/crio-89b65560e067827a311a7977d981827b7e986c13f9a998cc828eba30a11340d2 WatchSource:0}: Error finding container 89b65560e067827a311a7977d981827b7e986c13f9a998cc828eba30a11340d2: Status 404 returned error can't find the container with id 89b65560e067827a311a7977d981827b7e986c13f9a998cc828eba30a11340d2 Jul 15 14:42:05.097464 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:05.097428 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ts4lq" event={"ID":"e5a03076-1c4b-4cde-b963-862cdb07c9cd","Type":"ContainerStarted","Data":"89b65560e067827a311a7977d981827b7e986c13f9a998cc828eba30a11340d2"} Jul 15 14:42:07.111309 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:07.111274 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ts4lq" event={"ID":"e5a03076-1c4b-4cde-b963-862cdb07c9cd","Type":"ContainerStarted","Data":"d05ef3d5e3322f29e1c5cbc4e74543e1837d979cf1622fc9a8bc2c56580e8ecd"} Jul 15 14:42:07.111702 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:07.111349 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ts4lq" Jul 15 14:42:07.139829 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:07.139781 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ts4lq" podStartSLOduration=1.338477052 podStartE2EDuration="4.139765396s" totalImagesPullingTime="2.801288344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:42:07.138729888 +0000 UTC m=+499.844805992" watchObservedRunningTime="2026-07-15 14:42:07.139765396 +0000 UTC m=+499.845841492" Jul 15 14:42:18.117820 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:18.117782 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ts4lq" Jul 15 14:42:21.681933 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:21.681895 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq"] Jul 15 14:42:21.682312 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:21.682157 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" podUID="8600327f-7641-4e0c-b86a-69bb81aad3ca" containerName="manager" containerID="cri-o://9f1b0daf102d4f278979897c5ad16aa0a8ee24717d72ea68b8497a6d076e8a61" gracePeriod=2 Jul 15 14:42:21.691689 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:42:21.691654 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wxxhq\" is forbidden: User \"system:node:ip-10-0-139-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-83.ec2.internal' and this object" podUID="8600327f-7641-4e0c-b86a-69bb81aad3ca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" Jul 15 14:42:21.693381 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:21.693357 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq"] Jul 15 14:42:21.717428 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:21.717404 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctjjr"] Jul 15 14:42:21.717802 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:21.717786 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8600327f-7641-4e0c-b86a-69bb81aad3ca" containerName="manager" Jul 15 14:42:21.717802 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:21.717803 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8600327f-7641-4e0c-b86a-69bb81aad3ca" containerName="manager" Jul 15 14:42:21.717928 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:21.717858 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8600327f-7641-4e0c-b86a-69bb81aad3ca" containerName="manager" Jul 15 14:42:21.719685 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:21.719668 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctjjr" Jul 15 14:42:21.722119 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:42:21.722096 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wxxhq\" is forbidden: User \"system:node:ip-10-0-139-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-83.ec2.internal' and this object" podUID="8600327f-7641-4e0c-b86a-69bb81aad3ca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" Jul 15 14:42:21.734565 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:21.734539 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctjjr"] Jul 15 14:42:21.815726 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:21.815698 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnhwk\" (UniqueName: \"kubernetes.io/projected/c43ede6a-ac70-4059-aeb0-20513e15562e-kube-api-access-gnhwk\") pod \"limitador-operator-controller-manager-859d7b78bd-ctjjr\" (UID: \"c43ede6a-ac70-4059-aeb0-20513e15562e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctjjr" Jul 15 14:42:21.917052 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:21.917025 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnhwk\" (UniqueName: \"kubernetes.io/projected/c43ede6a-ac70-4059-aeb0-20513e15562e-kube-api-access-gnhwk\") pod \"limitador-operator-controller-manager-859d7b78bd-ctjjr\" (UID: \"c43ede6a-ac70-4059-aeb0-20513e15562e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctjjr" Jul 15 14:42:21.920970 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:21.920950 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" Jul 15 14:42:21.923546 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:42:21.923518 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wxxhq\" is forbidden: User \"system:node:ip-10-0-139-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-83.ec2.internal' and this object" podUID="8600327f-7641-4e0c-b86a-69bb81aad3ca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" Jul 15 14:42:21.928327 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:21.928302 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnhwk\" (UniqueName: \"kubernetes.io/projected/c43ede6a-ac70-4059-aeb0-20513e15562e-kube-api-access-gnhwk\") pod \"limitador-operator-controller-manager-859d7b78bd-ctjjr\" (UID: \"c43ede6a-ac70-4059-aeb0-20513e15562e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctjjr" Jul 15 14:42:22.018362 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:22.018304 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d69mx\" (UniqueName: \"kubernetes.io/projected/8600327f-7641-4e0c-b86a-69bb81aad3ca-kube-api-access-d69mx\") pod \"8600327f-7641-4e0c-b86a-69bb81aad3ca\" (UID: \"8600327f-7641-4e0c-b86a-69bb81aad3ca\") " Jul 15 14:42:22.020152 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:22.020131 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8600327f-7641-4e0c-b86a-69bb81aad3ca-kube-api-access-d69mx" (OuterVolumeSpecName: "kube-api-access-d69mx") pod "8600327f-7641-4e0c-b86a-69bb81aad3ca" (UID: "8600327f-7641-4e0c-b86a-69bb81aad3ca"). InnerVolumeSpecName "kube-api-access-d69mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:42:22.067423 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:22.067403 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctjjr" Jul 15 14:42:22.119490 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:22.119461 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d69mx\" (UniqueName: \"kubernetes.io/projected/8600327f-7641-4e0c-b86a-69bb81aad3ca-kube-api-access-d69mx\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:42:22.178014 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:22.177985 2574 generic.go:356] "Generic (PLEG): container finished" podID="8600327f-7641-4e0c-b86a-69bb81aad3ca" containerID="9f1b0daf102d4f278979897c5ad16aa0a8ee24717d72ea68b8497a6d076e8a61" exitCode=0 Jul 15 14:42:22.178176 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:22.178032 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" Jul 15 14:42:22.178176 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:22.178065 2574 scope.go:117] "RemoveContainer" containerID="9f1b0daf102d4f278979897c5ad16aa0a8ee24717d72ea68b8497a6d076e8a61" Jul 15 14:42:22.186261 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:42:22.186237 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wxxhq\" is forbidden: User \"system:node:ip-10-0-139-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-83.ec2.internal' and this object" podUID="8600327f-7641-4e0c-b86a-69bb81aad3ca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" Jul 15 14:42:22.188951 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:22.188934 2574 scope.go:117] "RemoveContainer" containerID="9f1b0daf102d4f278979897c5ad16aa0a8ee24717d72ea68b8497a6d076e8a61" Jul 15 14:42:22.189196 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:42:22.189179 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1b0daf102d4f278979897c5ad16aa0a8ee24717d72ea68b8497a6d076e8a61\": container with ID starting with 9f1b0daf102d4f278979897c5ad16aa0a8ee24717d72ea68b8497a6d076e8a61 not found: ID does not exist" containerID="9f1b0daf102d4f278979897c5ad16aa0a8ee24717d72ea68b8497a6d076e8a61" Jul 15 14:42:22.189246 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:22.189203 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1b0daf102d4f278979897c5ad16aa0a8ee24717d72ea68b8497a6d076e8a61"} err="failed to get container status \"9f1b0daf102d4f278979897c5ad16aa0a8ee24717d72ea68b8497a6d076e8a61\": rpc error: code = NotFound desc = could not find container \"9f1b0daf102d4f278979897c5ad16aa0a8ee24717d72ea68b8497a6d076e8a61\": container with ID starting with 9f1b0daf102d4f278979897c5ad16aa0a8ee24717d72ea68b8497a6d076e8a61 not found: ID does not exist" Jul 15 14:42:22.211304 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:42:22.211279 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wxxhq\" is forbidden: User \"system:node:ip-10-0-139-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-83.ec2.internal' and this object" podUID="8600327f-7641-4e0c-b86a-69bb81aad3ca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wxxhq" Jul 15 14:42:22.233178 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:22.233088 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctjjr"] Jul 15 14:42:22.235700 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:42:22.235680 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43ede6a_ac70_4059_aeb0_20513e15562e.slice/crio-e944fd3f70b02bee309cc6d7f58979cc042ded60ac634212969340e5928b2f39 WatchSource:0}: Error finding container e944fd3f70b02bee309cc6d7f58979cc042ded60ac634212969340e5928b2f39: Status 404 returned error can't find the container with id e944fd3f70b02bee309cc6d7f58979cc042ded60ac634212969340e5928b2f39 Jul 15 14:42:23.082978 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:23.082948 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8600327f-7641-4e0c-b86a-69bb81aad3ca" path="/var/lib/kubelet/pods/8600327f-7641-4e0c-b86a-69bb81aad3ca/volumes" Jul 15 14:42:23.183733 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:23.183699 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctjjr" event={"ID":"c43ede6a-ac70-4059-aeb0-20513e15562e","Type":"ContainerStarted","Data":"eca664d302acf8521b14539531117b1a11b349888d75a2428983c10522ae11be"} Jul 15 14:42:23.183943 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:23.183755 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctjjr" event={"ID":"c43ede6a-ac70-4059-aeb0-20513e15562e","Type":"ContainerStarted","Data":"e944fd3f70b02bee309cc6d7f58979cc042ded60ac634212969340e5928b2f39"} Jul 15 14:42:23.183943 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:23.183781 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctjjr" Jul 15 14:42:34.191529 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:34.191502 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctjjr" Jul 15 14:42:34.215273 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:34.215230 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ctjjr" podStartSLOduration=13.215217681 podStartE2EDuration="13.215217681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:42:23.213844841 +0000 UTC m=+515.919920946" watchObservedRunningTime="2026-07-15 14:42:34.215217681 +0000 UTC m=+526.921293787" Jul 15 14:42:50.234208 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.234181 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb"] Jul 15 14:42:50.242271 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.242253 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.247183 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.247164 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9w2rc\"" Jul 15 14:42:50.267847 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.267824 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb"] Jul 15 14:42:50.352410 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.352380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/008f284e-86e4-435f-8ab2-086bce5953a9-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.352540 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.352416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008f284e-86e4-435f-8ab2-086bce5953a9-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.352540 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.352448 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/008f284e-86e4-435f-8ab2-086bce5953a9-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.352540 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.352511 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/008f284e-86e4-435f-8ab2-086bce5953a9-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.352663 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.352539 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/008f284e-86e4-435f-8ab2-086bce5953a9-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.352663 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.352570 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/008f284e-86e4-435f-8ab2-086bce5953a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.352663 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.352598 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/008f284e-86e4-435f-8ab2-086bce5953a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.352795 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.352686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/008f284e-86e4-435f-8ab2-086bce5953a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.352795 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.352732 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9c2s\" (UniqueName: \"kubernetes.io/projected/008f284e-86e4-435f-8ab2-086bce5953a9-kube-api-access-m9c2s\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.453262 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.453232 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/008f284e-86e4-435f-8ab2-086bce5953a9-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.453397 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.453266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008f284e-86e4-435f-8ab2-086bce5953a9-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.453397 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.453297 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/008f284e-86e4-435f-8ab2-086bce5953a9-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.453397 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.453331 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/008f284e-86e4-435f-8ab2-086bce5953a9-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.453397 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.453356 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/008f284e-86e4-435f-8ab2-086bce5953a9-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.453397 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.453388 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/008f284e-86e4-435f-8ab2-086bce5953a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.453649 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.453430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/008f284e-86e4-435f-8ab2-086bce5953a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.453649 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.453485 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/008f284e-86e4-435f-8ab2-086bce5953a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.453649 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.453534 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9c2s\" (UniqueName: \"kubernetes.io/projected/008f284e-86e4-435f-8ab2-086bce5953a9-kube-api-access-m9c2s\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.453793 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.453663 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/008f284e-86e4-435f-8ab2-086bce5953a9-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.453793 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.453765 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/008f284e-86e4-435f-8ab2-086bce5953a9-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.453872 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.453797 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/008f284e-86e4-435f-8ab2-086bce5953a9-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.454032 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.454011 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/008f284e-86e4-435f-8ab2-086bce5953a9-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.454183 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.454157 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/008f284e-86e4-435f-8ab2-086bce5953a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.455721 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.455704 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/008f284e-86e4-435f-8ab2-086bce5953a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.455992 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.455974 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/008f284e-86e4-435f-8ab2-086bce5953a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.466616 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.466594 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/008f284e-86e4-435f-8ab2-086bce5953a9-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.466998 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.466982 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9c2s\" (UniqueName: \"kubernetes.io/projected/008f284e-86e4-435f-8ab2-086bce5953a9-kube-api-access-m9c2s\") pod \"maas-default-gateway-openshift-default-56654dcf7-ng2vb\" (UID: \"008f284e-86e4-435f-8ab2-086bce5953a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.552032 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.552002 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:50.703315 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.703284 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb"] Jul 15 14:42:50.709282 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:42:50.709252 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008f284e_86e4_435f_8ab2_086bce5953a9.slice/crio-0654b9900eee042ae300bf91483fc92ef8a0ece2ae9045ad2edba7c7b058480a WatchSource:0}: Error finding container 0654b9900eee042ae300bf91483fc92ef8a0ece2ae9045ad2edba7c7b058480a: Status 404 returned error can't find the container with id 0654b9900eee042ae300bf91483fc92ef8a0ece2ae9045ad2edba7c7b058480a Jul 15 14:42:50.711408 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.711371 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:42:50.711480 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.711452 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:42:50.711522 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:50.711486 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:42:51.312969 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:51.312931 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" event={"ID":"008f284e-86e4-435f-8ab2-086bce5953a9","Type":"ContainerStarted","Data":"fb3c6e784eb512b3c7ad721877ef3384873d3a66b5a3e8992b4bb895ef6fe10e"} Jul 15 14:42:51.313373 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:51.312984 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" event={"ID":"008f284e-86e4-435f-8ab2-086bce5953a9","Type":"ContainerStarted","Data":"0654b9900eee042ae300bf91483fc92ef8a0ece2ae9045ad2edba7c7b058480a"} Jul 15 14:42:51.347024 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:51.346977 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" podStartSLOduration=1.346931868 podStartE2EDuration="1.346931868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:42:51.345787259 +0000 UTC m=+544.051863364" watchObservedRunningTime="2026-07-15 14:42:51.346931868 +0000 UTC m=+544.053007972" Jul 15 14:42:51.553133 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:51.553096 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:51.557618 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:51.557590 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:52.317878 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:52.317845 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:42:52.318806 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:42:52.318784 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:43:08.147208 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.147177 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-wshpb"] Jul 15 14:43:08.150829 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.150813 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wshpb" Jul 15 14:43:08.153975 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.153945 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-856mb\"" Jul 15 14:43:08.158573 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.158554 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wshpb"] Jul 15 14:43:08.312222 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.312185 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbwxh\" (UniqueName: \"kubernetes.io/projected/83070ee6-af2b-4118-b8d5-d495faf8a882-kube-api-access-pbwxh\") pod \"authorino-58cd955675-wshpb\" (UID: \"83070ee6-af2b-4118-b8d5-d495faf8a882\") " pod="kuadrant-system/authorino-58cd955675-wshpb" Jul 15 14:43:08.413511 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.413427 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwxh\" (UniqueName: \"kubernetes.io/projected/83070ee6-af2b-4118-b8d5-d495faf8a882-kube-api-access-pbwxh\") pod \"authorino-58cd955675-wshpb\" (UID: \"83070ee6-af2b-4118-b8d5-d495faf8a882\") " pod="kuadrant-system/authorino-58cd955675-wshpb" Jul 15 14:43:08.424054 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.424031 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbwxh\" (UniqueName: \"kubernetes.io/projected/83070ee6-af2b-4118-b8d5-d495faf8a882-kube-api-access-pbwxh\") pod \"authorino-58cd955675-wshpb\" (UID: \"83070ee6-af2b-4118-b8d5-d495faf8a882\") " pod="kuadrant-system/authorino-58cd955675-wshpb" Jul 15 14:43:08.460989 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.460961 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wshpb" Jul 15 14:43:08.552051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.552019 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-pqm54"] Jul 15 14:43:08.556025 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.556004 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pqm54" Jul 15 14:43:08.570033 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.567798 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pqm54"] Jul 15 14:43:08.599963 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.599933 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-wshpb"] Jul 15 14:43:08.603577 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:43:08.603549 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83070ee6_af2b_4118_b8d5_d495faf8a882.slice/crio-0c2cf4da0adeb48087d32d7477f177ee1f75ff148d851507e67bf4684cbcab21 WatchSource:0}: Error finding container 0c2cf4da0adeb48087d32d7477f177ee1f75ff148d851507e67bf4684cbcab21: Status 404 returned error can't find the container with id 0c2cf4da0adeb48087d32d7477f177ee1f75ff148d851507e67bf4684cbcab21 Jul 15 14:43:08.716805 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.716718 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4db2\" (UniqueName: \"kubernetes.io/projected/9cd45fe6-647b-4a47-8001-be69d9a36447-kube-api-access-v4db2\") pod \"authorino-75d556bd77-pqm54\" (UID: \"9cd45fe6-647b-4a47-8001-be69d9a36447\") " pod="kuadrant-system/authorino-75d556bd77-pqm54" Jul 15 14:43:08.817496 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.817458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4db2\" (UniqueName: \"kubernetes.io/projected/9cd45fe6-647b-4a47-8001-be69d9a36447-kube-api-access-v4db2\") pod \"authorino-75d556bd77-pqm54\" (UID: \"9cd45fe6-647b-4a47-8001-be69d9a36447\") " pod="kuadrant-system/authorino-75d556bd77-pqm54" Jul 15 14:43:08.829777 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.829752 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4db2\" (UniqueName: \"kubernetes.io/projected/9cd45fe6-647b-4a47-8001-be69d9a36447-kube-api-access-v4db2\") pod \"authorino-75d556bd77-pqm54\" (UID: \"9cd45fe6-647b-4a47-8001-be69d9a36447\") " pod="kuadrant-system/authorino-75d556bd77-pqm54" Jul 15 14:43:08.876700 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:08.876653 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pqm54" Jul 15 14:43:09.015302 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:09.015279 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pqm54"] Jul 15 14:43:09.017495 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:43:09.017469 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cd45fe6_647b_4a47_8001_be69d9a36447.slice/crio-ee16c4f1a9c7a31e96905f6df2891e0ef58bb03aaa85be91d263ff7678271822 WatchSource:0}: Error finding container ee16c4f1a9c7a31e96905f6df2891e0ef58bb03aaa85be91d263ff7678271822: Status 404 returned error can't find the container with id ee16c4f1a9c7a31e96905f6df2891e0ef58bb03aaa85be91d263ff7678271822 Jul 15 14:43:09.394083 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:09.394024 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pqm54" event={"ID":"9cd45fe6-647b-4a47-8001-be69d9a36447","Type":"ContainerStarted","Data":"ee16c4f1a9c7a31e96905f6df2891e0ef58bb03aaa85be91d263ff7678271822"} Jul 15 14:43:09.395596 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:09.395528 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wshpb" event={"ID":"83070ee6-af2b-4118-b8d5-d495faf8a882","Type":"ContainerStarted","Data":"0c2cf4da0adeb48087d32d7477f177ee1f75ff148d851507e67bf4684cbcab21"} Jul 15 14:43:12.420089 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:12.420051 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pqm54" event={"ID":"9cd45fe6-647b-4a47-8001-be69d9a36447","Type":"ContainerStarted","Data":"cb60e7309e175bcc0ea3eac90ddf86141d620bd4425aa634680e0a822ed22eb1"} Jul 15 14:43:12.421543 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:12.421521 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wshpb" event={"ID":"83070ee6-af2b-4118-b8d5-d495faf8a882","Type":"ContainerStarted","Data":"2ef33625dd4003f16a9de57fab6ac6d2d631eda701ae7a35492cc68c0dee24c0"} Jul 15 14:43:12.477536 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:12.477489 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-pqm54" podStartSLOduration=1.531723637 podStartE2EDuration="4.477474268s" totalImagesPullingTime="2.945750631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:43:12.450557388 +0000 UTC m=+565.156633493" watchObservedRunningTime="2026-07-15 14:43:12.477474268 +0000 UTC m=+565.183550372" Jul 15 14:43:12.478592 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:12.478559 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-wshpb" podStartSLOduration=1.13037187 podStartE2EDuration="4.478549219s" totalImagesPullingTime="3.348177349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:43:12.477652914 +0000 UTC m=+565.183729020" watchObservedRunningTime="2026-07-15 14:43:12.478549219 +0000 UTC m=+565.184625324" Jul 15 14:43:12.528302 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:12.528261 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wshpb"] Jul 15 14:43:14.431548 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:14.431494 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-wshpb" podUID="83070ee6-af2b-4118-b8d5-d495faf8a882" containerName="authorino" containerID="cri-o://2ef33625dd4003f16a9de57fab6ac6d2d631eda701ae7a35492cc68c0dee24c0" gracePeriod=30 Jul 15 14:43:14.674291 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:14.674272 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wshpb" Jul 15 14:43:14.774004 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:14.773939 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbwxh\" (UniqueName: \"kubernetes.io/projected/83070ee6-af2b-4118-b8d5-d495faf8a882-kube-api-access-pbwxh\") pod \"83070ee6-af2b-4118-b8d5-d495faf8a882\" (UID: \"83070ee6-af2b-4118-b8d5-d495faf8a882\") " Jul 15 14:43:14.775823 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:14.775795 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83070ee6-af2b-4118-b8d5-d495faf8a882-kube-api-access-pbwxh" (OuterVolumeSpecName: "kube-api-access-pbwxh") pod "83070ee6-af2b-4118-b8d5-d495faf8a882" (UID: "83070ee6-af2b-4118-b8d5-d495faf8a882"). InnerVolumeSpecName "kube-api-access-pbwxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:43:14.874937 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:14.874912 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbwxh\" (UniqueName: \"kubernetes.io/projected/83070ee6-af2b-4118-b8d5-d495faf8a882-kube-api-access-pbwxh\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:43:15.436822 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:15.436734 2574 generic.go:356] "Generic (PLEG): container finished" podID="83070ee6-af2b-4118-b8d5-d495faf8a882" containerID="2ef33625dd4003f16a9de57fab6ac6d2d631eda701ae7a35492cc68c0dee24c0" exitCode=0 Jul 15 14:43:15.436822 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:15.436773 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-wshpb" Jul 15 14:43:15.436822 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:15.436790 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wshpb" event={"ID":"83070ee6-af2b-4118-b8d5-d495faf8a882","Type":"ContainerDied","Data":"2ef33625dd4003f16a9de57fab6ac6d2d631eda701ae7a35492cc68c0dee24c0"} Jul 15 14:43:15.437313 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:15.436830 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-wshpb" event={"ID":"83070ee6-af2b-4118-b8d5-d495faf8a882","Type":"ContainerDied","Data":"0c2cf4da0adeb48087d32d7477f177ee1f75ff148d851507e67bf4684cbcab21"} Jul 15 14:43:15.437313 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:15.436851 2574 scope.go:117] "RemoveContainer" containerID="2ef33625dd4003f16a9de57fab6ac6d2d631eda701ae7a35492cc68c0dee24c0" Jul 15 14:43:15.445539 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:15.445523 2574 scope.go:117] "RemoveContainer" containerID="2ef33625dd4003f16a9de57fab6ac6d2d631eda701ae7a35492cc68c0dee24c0" Jul 15 14:43:15.445779 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:43:15.445763 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef33625dd4003f16a9de57fab6ac6d2d631eda701ae7a35492cc68c0dee24c0\": container with ID starting with 2ef33625dd4003f16a9de57fab6ac6d2d631eda701ae7a35492cc68c0dee24c0 not found: ID does not exist" containerID="2ef33625dd4003f16a9de57fab6ac6d2d631eda701ae7a35492cc68c0dee24c0" Jul 15 14:43:15.445826 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:15.445787 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef33625dd4003f16a9de57fab6ac6d2d631eda701ae7a35492cc68c0dee24c0"} err="failed to get container status \"2ef33625dd4003f16a9de57fab6ac6d2d631eda701ae7a35492cc68c0dee24c0\": rpc error: code = NotFound desc = could not find container \"2ef33625dd4003f16a9de57fab6ac6d2d631eda701ae7a35492cc68c0dee24c0\": container with ID starting with 2ef33625dd4003f16a9de57fab6ac6d2d631eda701ae7a35492cc68c0dee24c0 not found: ID does not exist" Jul 15 14:43:15.456318 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:15.456299 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-wshpb"] Jul 15 14:43:15.461483 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:15.461464 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-wshpb"] Jul 15 14:43:17.083417 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:17.083387 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83070ee6-af2b-4118-b8d5-d495faf8a882" path="/var/lib/kubelet/pods/83070ee6-af2b-4118-b8d5-d495faf8a882/volumes" Jul 15 14:43:37.319084 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.319051 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ms2gr"] Jul 15 14:43:37.319560 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.319442 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83070ee6-af2b-4118-b8d5-d495faf8a882" containerName="authorino" Jul 15 14:43:37.319560 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.319459 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="83070ee6-af2b-4118-b8d5-d495faf8a882" containerName="authorino" Jul 15 14:43:37.319560 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.319544 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="83070ee6-af2b-4118-b8d5-d495faf8a882" containerName="authorino" Jul 15 14:43:37.323646 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.323627 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ms2gr" Jul 15 14:43:37.330778 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.330754 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ms2gr"] Jul 15 14:43:37.351059 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.351031 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n88bf\" (UniqueName: \"kubernetes.io/projected/3e0fc464-9d74-4a5b-9215-84c145b0f108-kube-api-access-n88bf\") pod \"authorino-5d4b67c446-ms2gr\" (UID: \"3e0fc464-9d74-4a5b-9215-84c145b0f108\") " pod="kuadrant-system/authorino-5d4b67c446-ms2gr" Jul 15 14:43:37.452471 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.452432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n88bf\" (UniqueName: \"kubernetes.io/projected/3e0fc464-9d74-4a5b-9215-84c145b0f108-kube-api-access-n88bf\") pod \"authorino-5d4b67c446-ms2gr\" (UID: \"3e0fc464-9d74-4a5b-9215-84c145b0f108\") " pod="kuadrant-system/authorino-5d4b67c446-ms2gr" Jul 15 14:43:37.463794 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.463762 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n88bf\" (UniqueName: \"kubernetes.io/projected/3e0fc464-9d74-4a5b-9215-84c145b0f108-kube-api-access-n88bf\") pod \"authorino-5d4b67c446-ms2gr\" (UID: \"3e0fc464-9d74-4a5b-9215-84c145b0f108\") " pod="kuadrant-system/authorino-5d4b67c446-ms2gr" Jul 15 14:43:37.633175 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.633103 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ms2gr" Jul 15 14:43:37.730728 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.730694 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ms2gr"] Jul 15 14:43:37.757061 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.757028 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5bc86d9d98-gc2lj"] Jul 15 14:43:37.759950 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.759932 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bc86d9d98-gc2lj" Jul 15 14:43:37.769582 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.769557 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ms2gr"] Jul 15 14:43:37.771226 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.771205 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5bc86d9d98-gc2lj"] Jul 15 14:43:37.771517 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:43:37.771494 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e0fc464_9d74_4a5b_9215_84c145b0f108.slice/crio-764d317f25c1d5b3cec5f4d2981deea0790f88be276db51d1ea6d79190c21c62 WatchSource:0}: Error finding container 764d317f25c1d5b3cec5f4d2981deea0790f88be276db51d1ea6d79190c21c62: Status 404 returned error can't find the container with id 764d317f25c1d5b3cec5f4d2981deea0790f88be276db51d1ea6d79190c21c62 Jul 15 14:43:37.856123 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.856092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2hhn\" (UniqueName: \"kubernetes.io/projected/ab6b2844-a04e-47a8-8a89-510757c73e2c-kube-api-access-p2hhn\") pod \"authorino-5bc86d9d98-gc2lj\" (UID: \"ab6b2844-a04e-47a8-8a89-510757c73e2c\") " pod="kuadrant-system/authorino-5bc86d9d98-gc2lj" Jul 15 14:43:37.957445 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.957363 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hhn\" (UniqueName: \"kubernetes.io/projected/ab6b2844-a04e-47a8-8a89-510757c73e2c-kube-api-access-p2hhn\") pod \"authorino-5bc86d9d98-gc2lj\" (UID: \"ab6b2844-a04e-47a8-8a89-510757c73e2c\") " pod="kuadrant-system/authorino-5bc86d9d98-gc2lj" Jul 15 14:43:37.971657 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:37.971633 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2hhn\" (UniqueName: \"kubernetes.io/projected/ab6b2844-a04e-47a8-8a89-510757c73e2c-kube-api-access-p2hhn\") pod \"authorino-5bc86d9d98-gc2lj\" (UID: \"ab6b2844-a04e-47a8-8a89-510757c73e2c\") " pod="kuadrant-system/authorino-5bc86d9d98-gc2lj" Jul 15 14:43:38.002123 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.002091 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5bc86d9d98-gc2lj"] Jul 15 14:43:38.002347 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.002335 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bc86d9d98-gc2lj" Jul 15 14:43:38.050414 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.050377 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d87cc4998-pjckd"] Jul 15 14:43:38.053897 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.053861 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d87cc4998-pjckd" Jul 15 14:43:38.056551 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.056531 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 15 14:43:38.064319 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.064293 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d87cc4998-pjckd"] Jul 15 14:43:38.138938 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.138914 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5bc86d9d98-gc2lj"] Jul 15 14:43:38.141937 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:43:38.141908 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab6b2844_a04e_47a8_8a89_510757c73e2c.slice/crio-e8b9d3b0b67f3730d0a46c9341a23efc1080d973c5699fd6f4f6753b92a43b40 WatchSource:0}: Error finding container e8b9d3b0b67f3730d0a46c9341a23efc1080d973c5699fd6f4f6753b92a43b40: Status 404 returned error can't find the container with id e8b9d3b0b67f3730d0a46c9341a23efc1080d973c5699fd6f4f6753b92a43b40 Jul 15 14:43:38.158534 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.158513 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf56574f-8ca5-4b6e-a60c-4efcac12d1d3-tls-cert\") pod \"authorino-d87cc4998-pjckd\" (UID: \"bf56574f-8ca5-4b6e-a60c-4efcac12d1d3\") " pod="kuadrant-system/authorino-d87cc4998-pjckd" Jul 15 14:43:38.158592 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.158559 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tfwv\" (UniqueName: \"kubernetes.io/projected/bf56574f-8ca5-4b6e-a60c-4efcac12d1d3-kube-api-access-4tfwv\") pod \"authorino-d87cc4998-pjckd\" (UID: \"bf56574f-8ca5-4b6e-a60c-4efcac12d1d3\") " pod="kuadrant-system/authorino-d87cc4998-pjckd" Jul 15 14:43:38.259054 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.258971 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfwv\" (UniqueName: \"kubernetes.io/projected/bf56574f-8ca5-4b6e-a60c-4efcac12d1d3-kube-api-access-4tfwv\") pod \"authorino-d87cc4998-pjckd\" (UID: \"bf56574f-8ca5-4b6e-a60c-4efcac12d1d3\") " pod="kuadrant-system/authorino-d87cc4998-pjckd" Jul 15 14:43:38.259218 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.259097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf56574f-8ca5-4b6e-a60c-4efcac12d1d3-tls-cert\") pod \"authorino-d87cc4998-pjckd\" (UID: \"bf56574f-8ca5-4b6e-a60c-4efcac12d1d3\") " pod="kuadrant-system/authorino-d87cc4998-pjckd" Jul 15 14:43:38.261454 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.261428 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf56574f-8ca5-4b6e-a60c-4efcac12d1d3-tls-cert\") pod \"authorino-d87cc4998-pjckd\" (UID: \"bf56574f-8ca5-4b6e-a60c-4efcac12d1d3\") " pod="kuadrant-system/authorino-d87cc4998-pjckd" Jul 15 14:43:38.268635 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.268613 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tfwv\" (UniqueName: \"kubernetes.io/projected/bf56574f-8ca5-4b6e-a60c-4efcac12d1d3-kube-api-access-4tfwv\") pod \"authorino-d87cc4998-pjckd\" (UID: \"bf56574f-8ca5-4b6e-a60c-4efcac12d1d3\") " pod="kuadrant-system/authorino-d87cc4998-pjckd" Jul 15 14:43:38.368677 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.368648 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d87cc4998-pjckd" Jul 15 14:43:38.497204 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.497169 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d87cc4998-pjckd"] Jul 15 14:43:38.497436 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:43:38.497410 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf56574f_8ca5_4b6e_a60c_4efcac12d1d3.slice/crio-ef401d9b9da0c1429f017fd035bd58859d414d42d832d5789a19d215e39f1a1f WatchSource:0}: Error finding container ef401d9b9da0c1429f017fd035bd58859d414d42d832d5789a19d215e39f1a1f: Status 404 returned error can't find the container with id ef401d9b9da0c1429f017fd035bd58859d414d42d832d5789a19d215e39f1a1f Jul 15 14:43:38.541980 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.541920 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d87cc4998-pjckd" event={"ID":"bf56574f-8ca5-4b6e-a60c-4efcac12d1d3","Type":"ContainerStarted","Data":"ef401d9b9da0c1429f017fd035bd58859d414d42d832d5789a19d215e39f1a1f"} Jul 15 14:43:38.543531 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.543498 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ms2gr" event={"ID":"3e0fc464-9d74-4a5b-9215-84c145b0f108","Type":"ContainerStarted","Data":"abc0f80f0fdcdaacc5001035bac5b45d7092c8574e8479bea4807b0f4e29de89"} Jul 15 14:43:38.543531 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.543515 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-ms2gr" podUID="3e0fc464-9d74-4a5b-9215-84c145b0f108" containerName="authorino" containerID="cri-o://abc0f80f0fdcdaacc5001035bac5b45d7092c8574e8479bea4807b0f4e29de89" gracePeriod=30 Jul 15 14:43:38.543679 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.543532 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ms2gr" event={"ID":"3e0fc464-9d74-4a5b-9215-84c145b0f108","Type":"ContainerStarted","Data":"764d317f25c1d5b3cec5f4d2981deea0790f88be276db51d1ea6d79190c21c62"} Jul 15 14:43:38.544757 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.544727 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bc86d9d98-gc2lj" event={"ID":"ab6b2844-a04e-47a8-8a89-510757c73e2c","Type":"ContainerStarted","Data":"e8b9d3b0b67f3730d0a46c9341a23efc1080d973c5699fd6f4f6753b92a43b40"} Jul 15 14:43:38.564941 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.564904 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-ms2gr" podStartSLOduration=1.224784161 podStartE2EDuration="1.564874763s" totalImagesPullingTime="340.090602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:43:38.562938802 +0000 UTC m=+591.269014907" watchObservedRunningTime="2026-07-15 14:43:38.564874763 +0000 UTC m=+591.270950868" Jul 15 14:43:38.786285 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.786260 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ms2gr" Jul 15 14:43:38.865196 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.865174 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n88bf\" (UniqueName: \"kubernetes.io/projected/3e0fc464-9d74-4a5b-9215-84c145b0f108-kube-api-access-n88bf\") pod \"3e0fc464-9d74-4a5b-9215-84c145b0f108\" (UID: \"3e0fc464-9d74-4a5b-9215-84c145b0f108\") " Jul 15 14:43:38.867102 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.867079 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0fc464-9d74-4a5b-9215-84c145b0f108-kube-api-access-n88bf" (OuterVolumeSpecName: "kube-api-access-n88bf") pod "3e0fc464-9d74-4a5b-9215-84c145b0f108" (UID: "3e0fc464-9d74-4a5b-9215-84c145b0f108"). InnerVolumeSpecName "kube-api-access-n88bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:43:38.966662 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:38.966639 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n88bf\" (UniqueName: \"kubernetes.io/projected/3e0fc464-9d74-4a5b-9215-84c145b0f108-kube-api-access-n88bf\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:43:39.556210 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.556174 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bc86d9d98-gc2lj" event={"ID":"ab6b2844-a04e-47a8-8a89-510757c73e2c","Type":"ContainerStarted","Data":"5b137138715567f8421fc05f4f6b6c20594f5a38c3895cbec41e5e06c7787ebf"} Jul 15 14:43:39.556635 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.556225 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5bc86d9d98-gc2lj" podUID="ab6b2844-a04e-47a8-8a89-510757c73e2c" containerName="authorino" containerID="cri-o://5b137138715567f8421fc05f4f6b6c20594f5a38c3895cbec41e5e06c7787ebf" gracePeriod=30 Jul 15 14:43:39.557481 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.557458 2574 generic.go:356] "Generic (PLEG): container finished" podID="3e0fc464-9d74-4a5b-9215-84c145b0f108" containerID="abc0f80f0fdcdaacc5001035bac5b45d7092c8574e8479bea4807b0f4e29de89" exitCode=0 Jul 15 14:43:39.557587 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.557517 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ms2gr" Jul 15 14:43:39.557587 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.557539 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ms2gr" event={"ID":"3e0fc464-9d74-4a5b-9215-84c145b0f108","Type":"ContainerDied","Data":"abc0f80f0fdcdaacc5001035bac5b45d7092c8574e8479bea4807b0f4e29de89"} Jul 15 14:43:39.557587 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.557578 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ms2gr" event={"ID":"3e0fc464-9d74-4a5b-9215-84c145b0f108","Type":"ContainerDied","Data":"764d317f25c1d5b3cec5f4d2981deea0790f88be276db51d1ea6d79190c21c62"} Jul 15 14:43:39.557693 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.557600 2574 scope.go:117] "RemoveContainer" containerID="abc0f80f0fdcdaacc5001035bac5b45d7092c8574e8479bea4807b0f4e29de89" Jul 15 14:43:39.566301 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.566281 2574 scope.go:117] "RemoveContainer" containerID="abc0f80f0fdcdaacc5001035bac5b45d7092c8574e8479bea4807b0f4e29de89" Jul 15 14:43:39.566528 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:43:39.566511 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abc0f80f0fdcdaacc5001035bac5b45d7092c8574e8479bea4807b0f4e29de89\": container with ID starting with abc0f80f0fdcdaacc5001035bac5b45d7092c8574e8479bea4807b0f4e29de89 not found: ID does not exist" containerID="abc0f80f0fdcdaacc5001035bac5b45d7092c8574e8479bea4807b0f4e29de89" Jul 15 14:43:39.566583 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.566533 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abc0f80f0fdcdaacc5001035bac5b45d7092c8574e8479bea4807b0f4e29de89"} err="failed to get container status \"abc0f80f0fdcdaacc5001035bac5b45d7092c8574e8479bea4807b0f4e29de89\": rpc error: code = NotFound desc = could not find container \"abc0f80f0fdcdaacc5001035bac5b45d7092c8574e8479bea4807b0f4e29de89\": container with ID starting with abc0f80f0fdcdaacc5001035bac5b45d7092c8574e8479bea4807b0f4e29de89 not found: ID does not exist" Jul 15 14:43:39.580340 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.580297 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5bc86d9d98-gc2lj" podStartSLOduration=2.180428661 podStartE2EDuration="2.580284543s" totalImagesPullingTime="399.855882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:43:39.578995218 +0000 UTC m=+592.285071322" watchObservedRunningTime="2026-07-15 14:43:39.580284543 +0000 UTC m=+592.286360686" Jul 15 14:43:39.608706 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.608684 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ms2gr"] Jul 15 14:43:39.613465 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.613444 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ms2gr"] Jul 15 14:43:39.797907 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.797857 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bc86d9d98-gc2lj" Jul 15 14:43:39.873064 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.873039 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2hhn\" (UniqueName: \"kubernetes.io/projected/ab6b2844-a04e-47a8-8a89-510757c73e2c-kube-api-access-p2hhn\") pod \"ab6b2844-a04e-47a8-8a89-510757c73e2c\" (UID: \"ab6b2844-a04e-47a8-8a89-510757c73e2c\") " Jul 15 14:43:39.875004 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.874983 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab6b2844-a04e-47a8-8a89-510757c73e2c-kube-api-access-p2hhn" (OuterVolumeSpecName: "kube-api-access-p2hhn") pod "ab6b2844-a04e-47a8-8a89-510757c73e2c" (UID: "ab6b2844-a04e-47a8-8a89-510757c73e2c"). InnerVolumeSpecName "kube-api-access-p2hhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:43:39.974047 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:39.973977 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2hhn\" (UniqueName: \"kubernetes.io/projected/ab6b2844-a04e-47a8-8a89-510757c73e2c-kube-api-access-p2hhn\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:43:40.563633 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.563599 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d87cc4998-pjckd" event={"ID":"bf56574f-8ca5-4b6e-a60c-4efcac12d1d3","Type":"ContainerStarted","Data":"14467f8596800c6dc03af12a7fad4cd1b0f30e7258cf5ef4e0d2bc2498a81e3c"} Jul 15 14:43:40.565615 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.565593 2574 generic.go:356] "Generic (PLEG): container finished" podID="ab6b2844-a04e-47a8-8a89-510757c73e2c" containerID="5b137138715567f8421fc05f4f6b6c20594f5a38c3895cbec41e5e06c7787ebf" exitCode=0 Jul 15 14:43:40.565751 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.565626 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5bc86d9d98-gc2lj" Jul 15 14:43:40.565751 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.565651 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bc86d9d98-gc2lj" event={"ID":"ab6b2844-a04e-47a8-8a89-510757c73e2c","Type":"ContainerDied","Data":"5b137138715567f8421fc05f4f6b6c20594f5a38c3895cbec41e5e06c7787ebf"} Jul 15 14:43:40.565751 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.565686 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5bc86d9d98-gc2lj" event={"ID":"ab6b2844-a04e-47a8-8a89-510757c73e2c","Type":"ContainerDied","Data":"e8b9d3b0b67f3730d0a46c9341a23efc1080d973c5699fd6f4f6753b92a43b40"} Jul 15 14:43:40.565751 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.565708 2574 scope.go:117] "RemoveContainer" containerID="5b137138715567f8421fc05f4f6b6c20594f5a38c3895cbec41e5e06c7787ebf" Jul 15 14:43:40.574412 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.574396 2574 scope.go:117] "RemoveContainer" containerID="5b137138715567f8421fc05f4f6b6c20594f5a38c3895cbec41e5e06c7787ebf" Jul 15 14:43:40.574661 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:43:40.574644 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b137138715567f8421fc05f4f6b6c20594f5a38c3895cbec41e5e06c7787ebf\": container with ID starting with 5b137138715567f8421fc05f4f6b6c20594f5a38c3895cbec41e5e06c7787ebf not found: ID does not exist" containerID="5b137138715567f8421fc05f4f6b6c20594f5a38c3895cbec41e5e06c7787ebf" Jul 15 14:43:40.574698 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.574671 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b137138715567f8421fc05f4f6b6c20594f5a38c3895cbec41e5e06c7787ebf"} err="failed to get container status \"5b137138715567f8421fc05f4f6b6c20594f5a38c3895cbec41e5e06c7787ebf\": rpc error: code = NotFound desc = could not find container \"5b137138715567f8421fc05f4f6b6c20594f5a38c3895cbec41e5e06c7787ebf\": container with ID starting with 5b137138715567f8421fc05f4f6b6c20594f5a38c3895cbec41e5e06c7787ebf not found: ID does not exist" Jul 15 14:43:40.588287 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.588245 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d87cc4998-pjckd" podStartSLOduration=0.820260938 podStartE2EDuration="2.588230736s" totalImagesPullingTime="1.767969798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:43:40.587195901 +0000 UTC m=+593.293272007" watchObservedRunningTime="2026-07-15 14:43:40.588230736 +0000 UTC m=+593.294306841" Jul 15 14:43:40.607252 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.607229 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5bc86d9d98-gc2lj"] Jul 15 14:43:40.611996 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.611972 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5bc86d9d98-gc2lj"] Jul 15 14:43:40.626253 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.626234 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pqm54"] Jul 15 14:43:40.626402 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.626382 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-pqm54" podUID="9cd45fe6-647b-4a47-8001-be69d9a36447" containerName="authorino" containerID="cri-o://cb60e7309e175bcc0ea3eac90ddf86141d620bd4425aa634680e0a822ed22eb1" gracePeriod=30 Jul 15 14:43:40.877058 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.877036 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pqm54" Jul 15 14:43:40.982207 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.982177 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4db2\" (UniqueName: \"kubernetes.io/projected/9cd45fe6-647b-4a47-8001-be69d9a36447-kube-api-access-v4db2\") pod \"9cd45fe6-647b-4a47-8001-be69d9a36447\" (UID: \"9cd45fe6-647b-4a47-8001-be69d9a36447\") " Jul 15 14:43:40.984163 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:40.984124 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd45fe6-647b-4a47-8001-be69d9a36447-kube-api-access-v4db2" (OuterVolumeSpecName: "kube-api-access-v4db2") pod "9cd45fe6-647b-4a47-8001-be69d9a36447" (UID: "9cd45fe6-647b-4a47-8001-be69d9a36447"). InnerVolumeSpecName "kube-api-access-v4db2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:43:41.083176 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:41.083148 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e0fc464-9d74-4a5b-9215-84c145b0f108" path="/var/lib/kubelet/pods/3e0fc464-9d74-4a5b-9215-84c145b0f108/volumes" Jul 15 14:43:41.083481 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:41.083468 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab6b2844-a04e-47a8-8a89-510757c73e2c" path="/var/lib/kubelet/pods/ab6b2844-a04e-47a8-8a89-510757c73e2c/volumes" Jul 15 14:43:41.083524 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:41.083499 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4db2\" (UniqueName: \"kubernetes.io/projected/9cd45fe6-647b-4a47-8001-be69d9a36447-kube-api-access-v4db2\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:43:41.571628 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:41.571597 2574 generic.go:356] "Generic (PLEG): container finished" podID="9cd45fe6-647b-4a47-8001-be69d9a36447" containerID="cb60e7309e175bcc0ea3eac90ddf86141d620bd4425aa634680e0a822ed22eb1" exitCode=0 Jul 15 14:43:41.572021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:41.571649 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pqm54" Jul 15 14:43:41.572021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:41.571654 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pqm54" event={"ID":"9cd45fe6-647b-4a47-8001-be69d9a36447","Type":"ContainerDied","Data":"cb60e7309e175bcc0ea3eac90ddf86141d620bd4425aa634680e0a822ed22eb1"} Jul 15 14:43:41.572021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:41.571688 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pqm54" event={"ID":"9cd45fe6-647b-4a47-8001-be69d9a36447","Type":"ContainerDied","Data":"ee16c4f1a9c7a31e96905f6df2891e0ef58bb03aaa85be91d263ff7678271822"} Jul 15 14:43:41.572021 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:41.571704 2574 scope.go:117] "RemoveContainer" containerID="cb60e7309e175bcc0ea3eac90ddf86141d620bd4425aa634680e0a822ed22eb1" Jul 15 14:43:41.580054 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:41.580024 2574 scope.go:117] "RemoveContainer" containerID="cb60e7309e175bcc0ea3eac90ddf86141d620bd4425aa634680e0a822ed22eb1" Jul 15 14:43:41.580322 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:43:41.580298 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb60e7309e175bcc0ea3eac90ddf86141d620bd4425aa634680e0a822ed22eb1\": container with ID starting with cb60e7309e175bcc0ea3eac90ddf86141d620bd4425aa634680e0a822ed22eb1 not found: ID does not exist" containerID="cb60e7309e175bcc0ea3eac90ddf86141d620bd4425aa634680e0a822ed22eb1" Jul 15 14:43:41.580371 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:41.580330 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb60e7309e175bcc0ea3eac90ddf86141d620bd4425aa634680e0a822ed22eb1"} err="failed to get container status \"cb60e7309e175bcc0ea3eac90ddf86141d620bd4425aa634680e0a822ed22eb1\": rpc error: code = NotFound desc = could not find container \"cb60e7309e175bcc0ea3eac90ddf86141d620bd4425aa634680e0a822ed22eb1\": container with ID starting with cb60e7309e175bcc0ea3eac90ddf86141d620bd4425aa634680e0a822ed22eb1 not found: ID does not exist" Jul 15 14:43:41.603711 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:41.603682 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pqm54"] Jul 15 14:43:41.605347 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:41.605331 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pqm54"] Jul 15 14:43:43.083767 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:43.083736 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd45fe6-647b-4a47-8001-be69d9a36447" path="/var/lib/kubelet/pods/9cd45fe6-647b-4a47-8001-be69d9a36447/volumes" Jul 15 14:43:48.878079 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:48.878053 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 14:43:48.879648 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:43:48.879623 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 14:44:30.799796 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:30.799710 2574 generic.go:356] "Generic (PLEG): container finished" podID="008f284e-86e4-435f-8ab2-086bce5953a9" containerID="fb3c6e784eb512b3c7ad721877ef3384873d3a66b5a3e8992b4bb895ef6fe10e" exitCode=0 Jul 15 14:44:30.800230 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:30.799796 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" event={"ID":"008f284e-86e4-435f-8ab2-086bce5953a9","Type":"ContainerDied","Data":"fb3c6e784eb512b3c7ad721877ef3384873d3a66b5a3e8992b4bb895ef6fe10e"} Jul 15 14:44:30.800279 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:30.800249 2574 scope.go:117] "RemoveContainer" containerID="fb3c6e784eb512b3c7ad721877ef3384873d3a66b5a3e8992b4bb895ef6fe10e" Jul 15 14:44:30.801247 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:30.801219 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:44:30.801303 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:30.801290 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:44:30.801339 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:30.801323 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 14:44:31.806153 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:31.806115 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" event={"ID":"008f284e-86e4-435f-8ab2-086bce5953a9","Type":"ContainerStarted","Data":"df1ae2c3140cf8aa19644f93606f76b32bd1b9f41a30cda919d2ae13765558ae"} Jul 15 14:44:32.553342 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:32.553302 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:44:32.553540 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:32.553424 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-ng2vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 15 14:44:32.553540 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:32.553470 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" podUID="008f284e-86e4-435f-8ab2-086bce5953a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 15 14:44:33.552575 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:33.552534 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-ng2vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 15 14:44:33.553058 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:33.552602 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" podUID="008f284e-86e4-435f-8ab2-086bce5953a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 15 14:44:34.553373 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:34.553333 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-ng2vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 15 14:44:34.553825 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:34.553412 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" podUID="008f284e-86e4-435f-8ab2-086bce5953a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 15 14:44:35.553008 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:35.552971 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-ng2vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 15 14:44:35.553292 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:35.553262 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" podUID="008f284e-86e4-435f-8ab2-086bce5953a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 15 14:44:36.557026 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:36.556992 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:44:36.557472 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:36.557407 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:44:36.558101 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:36.558080 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-ng2vb" Jul 15 14:44:45.698553 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.698521 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll"] Jul 15 14:44:45.699097 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.699045 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e0fc464-9d74-4a5b-9215-84c145b0f108" containerName="authorino" Jul 15 14:44:45.699097 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.699064 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e0fc464-9d74-4a5b-9215-84c145b0f108" containerName="authorino" Jul 15 14:44:45.699097 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.699090 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab6b2844-a04e-47a8-8a89-510757c73e2c" containerName="authorino" Jul 15 14:44:45.699265 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.699100 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab6b2844-a04e-47a8-8a89-510757c73e2c" containerName="authorino" Jul 15 14:44:45.699265 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.699125 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd45fe6-647b-4a47-8001-be69d9a36447" containerName="authorino" Jul 15 14:44:45.699265 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.699134 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd45fe6-647b-4a47-8001-be69d9a36447" containerName="authorino" Jul 15 14:44:45.699265 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.699213 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e0fc464-9d74-4a5b-9215-84c145b0f108" containerName="authorino" Jul 15 14:44:45.699265 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.699229 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd45fe6-647b-4a47-8001-be69d9a36447" containerName="authorino" Jul 15 14:44:45.699265 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.699240 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab6b2844-a04e-47a8-8a89-510757c73e2c" containerName="authorino" Jul 15 14:44:45.701073 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.701053 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.704732 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.704708 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 15 14:44:45.705994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.705952 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-68jzd\"" Jul 15 14:44:45.705994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.705952 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 14:44:45.705994 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.705992 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 14:44:45.715149 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.715128 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll"] Jul 15 14:44:45.826527 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.826500 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f419953-ed45-4208-bcb4-eece3355a9e3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.826655 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.826538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8tr\" (UniqueName: \"kubernetes.io/projected/2f419953-ed45-4208-bcb4-eece3355a9e3-kube-api-access-lx8tr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.826655 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.826558 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f419953-ed45-4208-bcb4-eece3355a9e3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.826655 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.826636 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f419953-ed45-4208-bcb4-eece3355a9e3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.826771 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.826669 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f419953-ed45-4208-bcb4-eece3355a9e3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.826771 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.826697 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f419953-ed45-4208-bcb4-eece3355a9e3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.927757 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.927725 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f419953-ed45-4208-bcb4-eece3355a9e3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.927757 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.927761 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f419953-ed45-4208-bcb4-eece3355a9e3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.928011 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.927798 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f419953-ed45-4208-bcb4-eece3355a9e3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.928011 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.927836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f419953-ed45-4208-bcb4-eece3355a9e3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.928011 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.927870 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8tr\" (UniqueName: \"kubernetes.io/projected/2f419953-ed45-4208-bcb4-eece3355a9e3-kube-api-access-lx8tr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.928011 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.927922 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f419953-ed45-4208-bcb4-eece3355a9e3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.928234 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.928163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f419953-ed45-4208-bcb4-eece3355a9e3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.928234 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.928199 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f419953-ed45-4208-bcb4-eece3355a9e3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.928338 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.928225 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f419953-ed45-4208-bcb4-eece3355a9e3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.930086 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.930059 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f419953-ed45-4208-bcb4-eece3355a9e3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.936346 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.936321 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f419953-ed45-4208-bcb4-eece3355a9e3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:45.941597 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:45.941577 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8tr\" (UniqueName: \"kubernetes.io/projected/2f419953-ed45-4208-bcb4-eece3355a9e3-kube-api-access-lx8tr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll\" (UID: \"2f419953-ed45-4208-bcb4-eece3355a9e3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:46.011985 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:46.011924 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:46.141572 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:46.141411 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll"] Jul 15 14:44:46.143628 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:44:46.143602 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f419953_ed45_4208_bcb4_eece3355a9e3.slice/crio-fb148af25cb82937de280cf5c08925f261c5e930f8a437efc0346bd91086e0e0 WatchSource:0}: Error finding container fb148af25cb82937de280cf5c08925f261c5e930f8a437efc0346bd91086e0e0: Status 404 returned error can't find the container with id fb148af25cb82937de280cf5c08925f261c5e930f8a437efc0346bd91086e0e0 Jul 15 14:44:46.144981 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:46.144965 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 14:44:46.880706 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:46.880665 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" event={"ID":"2f419953-ed45-4208-bcb4-eece3355a9e3","Type":"ContainerStarted","Data":"fb148af25cb82937de280cf5c08925f261c5e930f8a437efc0346bd91086e0e0"} Jul 15 14:44:47.886599 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:47.886565 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" event={"ID":"2f419953-ed45-4208-bcb4-eece3355a9e3","Type":"ContainerStarted","Data":"d2cd73996b91774e352412dbbf841649dc9de8ba73e081f0c2516f78c9f722c4"} Jul 15 14:44:47.909143 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:47.909099 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" podStartSLOduration=1.809759148 podStartE2EDuration="2.909081449s" totalImagesPullingTime="1.099322301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:44:47.907608552 +0000 UTC m=+660.613684651" watchObservedRunningTime="2026-07-15 14:44:47.909081449 +0000 UTC m=+660.615157556" Jul 15 14:44:56.012632 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:56.012584 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:56.013042 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:56.012654 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:56.024873 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:56.024844 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:44:56.939660 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:44:56.939629 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll" Jul 15 14:45:16.792546 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:16.792489 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg"] Jul 15 14:45:16.795618 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:16.795603 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:16.798309 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:16.798288 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 15 14:45:16.807051 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:16.807031 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg"] Jul 15 14:45:16.995334 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:16.995302 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:16.995518 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:16.995358 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:16.995518 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:16.995423 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:16.995518 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:16.995490 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:16.995670 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:16.995551 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:16.995670 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:16.995582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg727\" (UniqueName: \"kubernetes.io/projected/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-kube-api-access-kg727\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:17.096379 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:17.096348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:17.096546 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:17.096402 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:17.096546 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:17.096521 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:17.096671 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:17.096579 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg727\" (UniqueName: \"kubernetes.io/projected/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-kube-api-access-kg727\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:17.096671 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:17.096638 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:17.096782 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:17.096738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:17.096837 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:17.096738 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:17.097016 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:17.096995 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:17.097152 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:17.097132 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:17.098700 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:17.098672 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:17.099167 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:17.099148 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:17.110271 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:17.110247 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg727\" (UniqueName: \"kubernetes.io/projected/ec4e88f3-533d-40f3-bf2f-0d7cde309ddd-kube-api-access-kg727\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg\" (UID: \"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:17.405720 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:17.405639 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:17.542271 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:17.542246 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg"] Jul 15 14:45:17.544042 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:45:17.544010 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4e88f3_533d_40f3_bf2f_0d7cde309ddd.slice/crio-4fcdccd286587839cb170e22f8d56b6ecce2637b2d4f79d154c216420bff0f40 WatchSource:0}: Error finding container 4fcdccd286587839cb170e22f8d56b6ecce2637b2d4f79d154c216420bff0f40: Status 404 returned error can't find the container with id 4fcdccd286587839cb170e22f8d56b6ecce2637b2d4f79d154c216420bff0f40 Jul 15 14:45:18.032539 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:18.032503 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" event={"ID":"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd","Type":"ContainerStarted","Data":"4fcdccd286587839cb170e22f8d56b6ecce2637b2d4f79d154c216420bff0f40"} Jul 15 14:45:19.044081 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:19.044042 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" event={"ID":"ec4e88f3-533d-40f3-bf2f-0d7cde309ddd","Type":"ContainerStarted","Data":"9d3f7e51709436ad1637babc52a1aafb7abfb04facd1fa97d5f864d83fb07091"} Jul 15 14:45:19.072552 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:19.072499 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" podStartSLOduration=2.509859418 podStartE2EDuration="3.07248335s" totalImagesPullingTime="562.623932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:45:19.071626031 +0000 UTC m=+691.777702137" watchObservedRunningTime="2026-07-15 14:45:19.07248335 +0000 UTC m=+691.778559454" Jul 15 14:45:27.406280 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:27.406225 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:27.406652 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:27.406307 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:27.418713 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:27.418690 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:28.100115 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:28.100085 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg" Jul 15 14:45:33.482486 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.482457 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj"] Jul 15 14:45:33.485827 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.485810 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.489547 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.489529 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 15 14:45:33.503044 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.503025 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj"] Jul 15 14:45:33.537829 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.537810 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89395a8a-a38d-4b7a-b059-4419aac9ca1d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.537954 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.537837 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89395a8a-a38d-4b7a-b059-4419aac9ca1d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.537954 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.537860 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8jtm\" (UniqueName: \"kubernetes.io/projected/89395a8a-a38d-4b7a-b059-4419aac9ca1d-kube-api-access-r8jtm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.537954 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.537932 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89395a8a-a38d-4b7a-b059-4419aac9ca1d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.538057 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.537984 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89395a8a-a38d-4b7a-b059-4419aac9ca1d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.538057 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.538022 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89395a8a-a38d-4b7a-b059-4419aac9ca1d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.638980 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.638958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89395a8a-a38d-4b7a-b059-4419aac9ca1d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.639075 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.638994 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89395a8a-a38d-4b7a-b059-4419aac9ca1d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.639075 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.639016 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89395a8a-a38d-4b7a-b059-4419aac9ca1d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.639075 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.639048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8jtm\" (UniqueName: \"kubernetes.io/projected/89395a8a-a38d-4b7a-b059-4419aac9ca1d-kube-api-access-r8jtm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.639205 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.639093 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89395a8a-a38d-4b7a-b059-4419aac9ca1d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.639205 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.639132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89395a8a-a38d-4b7a-b059-4419aac9ca1d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.639359 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.639332 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89395a8a-a38d-4b7a-b059-4419aac9ca1d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.639473 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.639390 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89395a8a-a38d-4b7a-b059-4419aac9ca1d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.639520 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.639480 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89395a8a-a38d-4b7a-b059-4419aac9ca1d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.641226 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.641203 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89395a8a-a38d-4b7a-b059-4419aac9ca1d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.641471 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.641454 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89395a8a-a38d-4b7a-b059-4419aac9ca1d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.653754 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.653726 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8jtm\" (UniqueName: \"kubernetes.io/projected/89395a8a-a38d-4b7a-b059-4419aac9ca1d-kube-api-access-r8jtm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj\" (UID: \"89395a8a-a38d-4b7a-b059-4419aac9ca1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.795282 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.795187 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:33.936420 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:33.936395 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj"] Jul 15 14:45:33.938148 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:45:33.938105 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89395a8a_a38d_4b7a_b059_4419aac9ca1d.slice/crio-b66fc920b82218206174a78a8e1a391aa01a6b3d5c81efb69d0a4253ecc528f6 WatchSource:0}: Error finding container b66fc920b82218206174a78a8e1a391aa01a6b3d5c81efb69d0a4253ecc528f6: Status 404 returned error can't find the container with id b66fc920b82218206174a78a8e1a391aa01a6b3d5c81efb69d0a4253ecc528f6 Jul 15 14:45:34.118354 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:34.118313 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" event={"ID":"89395a8a-a38d-4b7a-b059-4419aac9ca1d","Type":"ContainerStarted","Data":"b66fc920b82218206174a78a8e1a391aa01a6b3d5c81efb69d0a4253ecc528f6"} Jul 15 14:45:35.124185 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:35.124133 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" event={"ID":"89395a8a-a38d-4b7a-b059-4419aac9ca1d","Type":"ContainerStarted","Data":"349e52dece97b5ed6ef76a06ce4d79deec5bb232698e59c6c25ca002b27eb744"} Jul 15 14:45:35.147374 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:35.147295 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" podStartSLOduration=1.856818257 podStartE2EDuration="2.147277028s" totalImagesPullingTime="290.458771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:45:35.146272596 +0000 UTC m=+707.852348704" watchObservedRunningTime="2026-07-15 14:45:35.147277028 +0000 UTC m=+707.853353135" Jul 15 14:45:43.796135 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:43.796049 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:43.796135 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:43.796096 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:43.809089 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:43.809062 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:45:44.178203 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:45:44.178175 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj" Jul 15 14:46:34.765113 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:34.765077 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d449b7fcf-9zf46"] Jul 15 14:46:34.770224 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:34.770180 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d449b7fcf-9zf46" Jul 15 14:46:34.776666 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:34.776640 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d449b7fcf-9zf46"] Jul 15 14:46:34.874901 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:34.874853 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a89662fc-7a0b-40c9-8c3b-34483f6be84c-tls-cert\") pod \"authorino-5d449b7fcf-9zf46\" (UID: \"a89662fc-7a0b-40c9-8c3b-34483f6be84c\") " pod="kuadrant-system/authorino-5d449b7fcf-9zf46" Jul 15 14:46:34.875064 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:34.874920 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k44fj\" (UniqueName: \"kubernetes.io/projected/a89662fc-7a0b-40c9-8c3b-34483f6be84c-kube-api-access-k44fj\") pod \"authorino-5d449b7fcf-9zf46\" (UID: \"a89662fc-7a0b-40c9-8c3b-34483f6be84c\") " pod="kuadrant-system/authorino-5d449b7fcf-9zf46" Jul 15 14:46:34.975453 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:34.975419 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a89662fc-7a0b-40c9-8c3b-34483f6be84c-tls-cert\") pod \"authorino-5d449b7fcf-9zf46\" (UID: \"a89662fc-7a0b-40c9-8c3b-34483f6be84c\") " pod="kuadrant-system/authorino-5d449b7fcf-9zf46" Jul 15 14:46:34.975635 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:34.975465 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k44fj\" (UniqueName: \"kubernetes.io/projected/a89662fc-7a0b-40c9-8c3b-34483f6be84c-kube-api-access-k44fj\") pod \"authorino-5d449b7fcf-9zf46\" (UID: \"a89662fc-7a0b-40c9-8c3b-34483f6be84c\") " pod="kuadrant-system/authorino-5d449b7fcf-9zf46" Jul 15 14:46:34.977853 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:34.977827 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a89662fc-7a0b-40c9-8c3b-34483f6be84c-tls-cert\") pod \"authorino-5d449b7fcf-9zf46\" (UID: \"a89662fc-7a0b-40c9-8c3b-34483f6be84c\") " pod="kuadrant-system/authorino-5d449b7fcf-9zf46" Jul 15 14:46:34.991523 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:34.991495 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k44fj\" (UniqueName: \"kubernetes.io/projected/a89662fc-7a0b-40c9-8c3b-34483f6be84c-kube-api-access-k44fj\") pod \"authorino-5d449b7fcf-9zf46\" (UID: \"a89662fc-7a0b-40c9-8c3b-34483f6be84c\") " pod="kuadrant-system/authorino-5d449b7fcf-9zf46" Jul 15 14:46:35.080715 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:35.080684 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d449b7fcf-9zf46" Jul 15 14:46:35.219860 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:35.219822 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d449b7fcf-9zf46"] Jul 15 14:46:35.223200 ip-10-0-139-83 kubenswrapper[2574]: W0715 14:46:35.223168 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda89662fc_7a0b_40c9_8c3b_34483f6be84c.slice/crio-4ba69ad1a226fe804f1c240fde25c2bce4f6eec367dc431d246efd4d30dfb8a5 WatchSource:0}: Error finding container 4ba69ad1a226fe804f1c240fde25c2bce4f6eec367dc431d246efd4d30dfb8a5: Status 404 returned error can't find the container with id 4ba69ad1a226fe804f1c240fde25c2bce4f6eec367dc431d246efd4d30dfb8a5 Jul 15 14:46:35.403716 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:35.403628 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d449b7fcf-9zf46" event={"ID":"a89662fc-7a0b-40c9-8c3b-34483f6be84c","Type":"ContainerStarted","Data":"4ba69ad1a226fe804f1c240fde25c2bce4f6eec367dc431d246efd4d30dfb8a5"} Jul 15 14:46:36.410519 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:36.410485 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d449b7fcf-9zf46" event={"ID":"a89662fc-7a0b-40c9-8c3b-34483f6be84c","Type":"ContainerStarted","Data":"901163b2994a7d5422e286542c2d527ac20658a267d8c8971aeb686ae189fdee"} Jul 15 14:46:36.443941 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:36.443872 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d449b7fcf-9zf46" podStartSLOduration=1.9973240410000002 podStartE2EDuration="2.443850766s" totalImagesPullingTime="446.526725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 14:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 14:46:36.441943379 +0000 UTC m=+769.148019486" watchObservedRunningTime="2026-07-15 14:46:36.443850766 +0000 UTC m=+769.149926872" Jul 15 14:46:36.483270 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:36.483227 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d87cc4998-pjckd"] Jul 15 14:46:36.483569 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:36.483536 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-d87cc4998-pjckd" podUID="bf56574f-8ca5-4b6e-a60c-4efcac12d1d3" containerName="authorino" containerID="cri-o://14467f8596800c6dc03af12a7fad4cd1b0f30e7258cf5ef4e0d2bc2498a81e3c" gracePeriod=30 Jul 15 14:46:36.748183 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:36.748147 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d87cc4998-pjckd" Jul 15 14:46:36.893880 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:36.893847 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tfwv\" (UniqueName: \"kubernetes.io/projected/bf56574f-8ca5-4b6e-a60c-4efcac12d1d3-kube-api-access-4tfwv\") pod \"bf56574f-8ca5-4b6e-a60c-4efcac12d1d3\" (UID: \"bf56574f-8ca5-4b6e-a60c-4efcac12d1d3\") " Jul 15 14:46:36.894078 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:36.893914 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf56574f-8ca5-4b6e-a60c-4efcac12d1d3-tls-cert\") pod \"bf56574f-8ca5-4b6e-a60c-4efcac12d1d3\" (UID: \"bf56574f-8ca5-4b6e-a60c-4efcac12d1d3\") " Jul 15 14:46:36.896087 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:36.896056 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf56574f-8ca5-4b6e-a60c-4efcac12d1d3-kube-api-access-4tfwv" (OuterVolumeSpecName: "kube-api-access-4tfwv") pod "bf56574f-8ca5-4b6e-a60c-4efcac12d1d3" (UID: "bf56574f-8ca5-4b6e-a60c-4efcac12d1d3"). InnerVolumeSpecName "kube-api-access-4tfwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 14:46:36.904757 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:36.904723 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf56574f-8ca5-4b6e-a60c-4efcac12d1d3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "bf56574f-8ca5-4b6e-a60c-4efcac12d1d3" (UID: "bf56574f-8ca5-4b6e-a60c-4efcac12d1d3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 14:46:36.995770 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:36.995736 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tfwv\" (UniqueName: \"kubernetes.io/projected/bf56574f-8ca5-4b6e-a60c-4efcac12d1d3-kube-api-access-4tfwv\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:46:36.995770 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:36.995769 2574 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf56574f-8ca5-4b6e-a60c-4efcac12d1d3-tls-cert\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 14:46:37.417309 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:37.417267 2574 generic.go:356] "Generic (PLEG): container finished" podID="bf56574f-8ca5-4b6e-a60c-4efcac12d1d3" containerID="14467f8596800c6dc03af12a7fad4cd1b0f30e7258cf5ef4e0d2bc2498a81e3c" exitCode=0 Jul 15 14:46:37.417309 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:37.417305 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d87cc4998-pjckd" Jul 15 14:46:37.417826 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:37.417345 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d87cc4998-pjckd" event={"ID":"bf56574f-8ca5-4b6e-a60c-4efcac12d1d3","Type":"ContainerDied","Data":"14467f8596800c6dc03af12a7fad4cd1b0f30e7258cf5ef4e0d2bc2498a81e3c"} Jul 15 14:46:37.417826 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:37.417387 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d87cc4998-pjckd" event={"ID":"bf56574f-8ca5-4b6e-a60c-4efcac12d1d3","Type":"ContainerDied","Data":"ef401d9b9da0c1429f017fd035bd58859d414d42d832d5789a19d215e39f1a1f"} Jul 15 14:46:37.417826 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:37.417404 2574 scope.go:117] "RemoveContainer" containerID="14467f8596800c6dc03af12a7fad4cd1b0f30e7258cf5ef4e0d2bc2498a81e3c" Jul 15 14:46:37.426168 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:37.426141 2574 scope.go:117] "RemoveContainer" containerID="14467f8596800c6dc03af12a7fad4cd1b0f30e7258cf5ef4e0d2bc2498a81e3c" Jul 15 14:46:37.426440 ip-10-0-139-83 kubenswrapper[2574]: E0715 14:46:37.426422 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14467f8596800c6dc03af12a7fad4cd1b0f30e7258cf5ef4e0d2bc2498a81e3c\": container with ID starting with 14467f8596800c6dc03af12a7fad4cd1b0f30e7258cf5ef4e0d2bc2498a81e3c not found: ID does not exist" containerID="14467f8596800c6dc03af12a7fad4cd1b0f30e7258cf5ef4e0d2bc2498a81e3c" Jul 15 14:46:37.426509 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:37.426447 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14467f8596800c6dc03af12a7fad4cd1b0f30e7258cf5ef4e0d2bc2498a81e3c"} err="failed to get container status \"14467f8596800c6dc03af12a7fad4cd1b0f30e7258cf5ef4e0d2bc2498a81e3c\": rpc error: code = NotFound desc = could not find container \"14467f8596800c6dc03af12a7fad4cd1b0f30e7258cf5ef4e0d2bc2498a81e3c\": container with ID starting with 14467f8596800c6dc03af12a7fad4cd1b0f30e7258cf5ef4e0d2bc2498a81e3c not found: ID does not exist" Jul 15 14:46:37.438645 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:37.438603 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d87cc4998-pjckd"] Jul 15 14:46:37.441920 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:37.441873 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d87cc4998-pjckd"] Jul 15 14:46:39.084168 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:46:39.084137 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf56574f-8ca5-4b6e-a60c-4efcac12d1d3" path="/var/lib/kubelet/pods/bf56574f-8ca5-4b6e-a60c-4efcac12d1d3/volumes" Jul 15 14:48:48.910201 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:48:48.910168 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 14:48:48.910201 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:48:48.910201 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 14:53:48.938586 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:53:48.938555 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 14:53:48.940529 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:53:48.940504 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 14:58:48.968671 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:58:48.968645 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 14:58:48.971372 ip-10-0-139-83 kubenswrapper[2574]: I0715 14:58:48.971354 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 15:00:13.541971 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.541936 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm"] Jul 15 15:00:13.542430 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.542327 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf56574f-8ca5-4b6e-a60c-4efcac12d1d3" containerName="authorino" Jul 15 15:00:13.542430 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.542348 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf56574f-8ca5-4b6e-a60c-4efcac12d1d3" containerName="authorino" Jul 15 15:00:13.542550 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.542482 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf56574f-8ca5-4b6e-a60c-4efcac12d1d3" containerName="authorino" Jul 15 15:00:13.551205 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.550927 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.553680 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.553655 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8a5fa064-openshift-default-dockercfg-66qqr\"" Jul 15 15:00:13.560784 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.560761 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm"] Jul 15 15:00:13.646364 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.646336 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-envoy\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.646486 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.646367 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-data\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.646486 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.646387 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-credential-socket\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.646486 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.646407 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5dn2\" (UniqueName: \"kubernetes.io/projected/124154bd-e2bf-41c9-ba1a-1a788c82c654-kube-api-access-g5dn2\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.646486 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.646428 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124154bd-e2bf-41c9-ba1a-1a788c82c654-istiod-ca-cert\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.646636 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.646487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-workload-certs\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.646636 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.646516 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-workload-socket\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.646636 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.646530 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-token\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.646636 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.646549 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-podinfo\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.747341 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.747308 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5dn2\" (UniqueName: \"kubernetes.io/projected/124154bd-e2bf-41c9-ba1a-1a788c82c654-kube-api-access-g5dn2\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.747471 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.747345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124154bd-e2bf-41c9-ba1a-1a788c82c654-istiod-ca-cert\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.747471 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.747389 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-workload-certs\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.747471 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.747426 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-workload-socket\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.747471 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.747442 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-token\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.747688 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.747556 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-podinfo\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.747688 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.747621 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-envoy\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.747688 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.747648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-data\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.747688 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.747676 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-credential-socket\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.747909 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.747823 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-workload-certs\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.748037 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.748008 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-credential-socket\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.748152 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.748080 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-workload-socket\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.748152 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.748093 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-data\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.748266 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.748231 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124154bd-e2bf-41c9-ba1a-1a788c82c654-istiod-ca-cert\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.749782 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.749761 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-envoy\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.750111 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.750090 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-podinfo\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.755476 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.755457 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-token\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.755712 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.755696 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5dn2\" (UniqueName: \"kubernetes.io/projected/124154bd-e2bf-41c9-ba1a-1a788c82c654-kube-api-access-g5dn2\") pod \"e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.861748 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.861723 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:13.991180 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.991152 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm"] Jul 15 15:00:13.993813 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:00:13.993775 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124154bd_e2bf_41c9_ba1a_1a788c82c654.slice/crio-016eaf605331bf9a4f5ad82f3e96a52ba6794e458e720d5ba9f6d8cdee78a8ba WatchSource:0}: Error finding container 016eaf605331bf9a4f5ad82f3e96a52ba6794e458e720d5ba9f6d8cdee78a8ba: Status 404 returned error can't find the container with id 016eaf605331bf9a4f5ad82f3e96a52ba6794e458e720d5ba9f6d8cdee78a8ba Jul 15 15:00:13.995981 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.995948 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:00:13.996047 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.996013 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:00:13.996047 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:13.996038 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:00:14.263468 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:14.263387 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" event={"ID":"124154bd-e2bf-41c9-ba1a-1a788c82c654","Type":"ContainerStarted","Data":"5bcc5bb2ee3c781a447b2d597b33f192969675c1591a2c8cbdf2d0d65545d9df"} Jul 15 15:00:14.263468 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:14.263440 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" event={"ID":"124154bd-e2bf-41c9-ba1a-1a788c82c654","Type":"ContainerStarted","Data":"016eaf605331bf9a4f5ad82f3e96a52ba6794e458e720d5ba9f6d8cdee78a8ba"} Jul 15 15:00:14.285432 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:14.285385 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" podStartSLOduration=1.285370631 podStartE2EDuration="1.285370631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:00:14.28346619 +0000 UTC m=+1586.989542293" watchObservedRunningTime="2026-07-15 15:00:14.285370631 +0000 UTC m=+1586.991446734" Jul 15 15:00:14.862346 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:14.862314 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:15.866750 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:15.866715 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:16.272707 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:16.272636 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:16.273600 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:16.273584 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:26.319340 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:26.319309 2574 generic.go:356] "Generic (PLEG): container finished" podID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerID="5bcc5bb2ee3c781a447b2d597b33f192969675c1591a2c8cbdf2d0d65545d9df" exitCode=0 Jul 15 15:00:26.319825 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:26.319381 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" event={"ID":"124154bd-e2bf-41c9-ba1a-1a788c82c654","Type":"ContainerDied","Data":"5bcc5bb2ee3c781a447b2d597b33f192969675c1591a2c8cbdf2d0d65545d9df"} Jul 15 15:00:26.319825 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:26.319698 2574 scope.go:117] "RemoveContainer" containerID="5bcc5bb2ee3c781a447b2d597b33f192969675c1591a2c8cbdf2d0d65545d9df" Jul 15 15:00:26.320531 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:26.320504 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:00:26.320613 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:26.320567 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:00:26.320613 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:26.320600 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:00:27.334217 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:27.334182 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" event={"ID":"124154bd-e2bf-41c9-ba1a-1a788c82c654","Type":"ContainerStarted","Data":"3579b2f055e4a0ab855788d7c433a517abd156b899a619b5e09f3a3c6cc51802"} Jul 15 15:00:27.862667 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:27.862632 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:27.863555 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:27.863528 2574 patch_prober.go:28] interesting pod/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 15:00:27.863659 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:27.863588 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" podUID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 15:00:28.862193 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:28.862158 2574 patch_prober.go:28] interesting pod/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 15:00:28.862625 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:28.862219 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" podUID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 15:00:29.862978 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:29.862935 2574 patch_prober.go:28] interesting pod/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 15:00:29.863462 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:29.863018 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" podUID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 15:00:30.862471 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:30.862427 2574 patch_prober.go:28] interesting pod/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 15:00:30.862662 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:30.862493 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" podUID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 15:00:31.272778 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:31.272688 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:31.863172 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:31.863131 2574 patch_prober.go:28] interesting pod/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 15:00:31.863461 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:31.863433 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" podUID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 15:00:32.862683 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:32.862646 2574 patch_prober.go:28] interesting pod/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 15 15:00:32.863277 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:32.863244 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" podUID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 15 15:00:33.865428 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:33.865397 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:00:33.866422 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:00:33.866403 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:01:45.228761 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:45.228721 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm"] Jul 15 15:01:45.229213 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:45.228995 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" podUID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerName="istio-proxy" containerID="cri-o://3579b2f055e4a0ab855788d7c433a517abd156b899a619b5e09f3a3c6cc51802" gracePeriod=30 Jul 15 15:01:45.819695 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:45.819660 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7"] Jul 15 15:01:45.822631 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:45.822605 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:45.825289 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:45.825265 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-fc2c3cfa-gw-openshift-default-dockercfg-bzg28\"" Jul 15 15:01:45.836439 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:45.836404 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7"] Jul 15 15:01:45.930388 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:45.930354 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-credential-socket\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:45.930388 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:45.930391 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-data\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:45.930644 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:45.930414 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7fbh\" (UniqueName: \"kubernetes.io/projected/612d082a-4654-4fd7-b9ce-483cd5444f00-kube-api-access-x7fbh\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:45.930644 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:45.930500 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-podinfo\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:45.930644 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:45.930532 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612d082a-4654-4fd7-b9ce-483cd5444f00-istiod-ca-cert\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:45.930644 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:45.930558 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-token\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:45.930842 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:45.930643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-workload-socket\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:45.930842 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:45.930726 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-workload-certs\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:45.930842 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:45.930768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-envoy\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.031303 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.031258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-podinfo\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.031303 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.031303 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612d082a-4654-4fd7-b9ce-483cd5444f00-istiod-ca-cert\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.031578 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.031324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-token\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.031578 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.031417 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-workload-socket\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.031578 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.031484 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-workload-certs\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.031578 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.031523 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-envoy\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.031793 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.031597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-credential-socket\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.031793 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.031626 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-data\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.031793 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.031728 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7fbh\" (UniqueName: \"kubernetes.io/projected/612d082a-4654-4fd7-b9ce-483cd5444f00-kube-api-access-x7fbh\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.031979 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.031923 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-workload-certs\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.031979 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.031935 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-workload-socket\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.032123 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.032088 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-credential-socket\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.032182 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.032128 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-data\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.032260 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.032240 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612d082a-4654-4fd7-b9ce-483cd5444f00-istiod-ca-cert\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.034037 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.034012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-podinfo\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.034151 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.034118 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-envoy\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.044369 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.044335 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-token\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.044565 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.044541 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7fbh\" (UniqueName: \"kubernetes.io/projected/612d082a-4654-4fd7-b9ce-483cd5444f00-kube-api-access-x7fbh\") pod \"e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.136659 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.136560 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:46.275124 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.275096 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7"] Jul 15 15:01:46.277569 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:01:46.277546 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612d082a_4654_4fd7_b9ce_483cd5444f00.slice/crio-8dbdf36673459935920f40ed9d77b48499c66e69bdd451cdd9eca4ba04fcfeff WatchSource:0}: Error finding container 8dbdf36673459935920f40ed9d77b48499c66e69bdd451cdd9eca4ba04fcfeff: Status 404 returned error can't find the container with id 8dbdf36673459935920f40ed9d77b48499c66e69bdd451cdd9eca4ba04fcfeff Jul 15 15:01:46.279659 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.279626 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:01:46.279709 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.279696 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:01:46.279753 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.279722 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:01:46.725402 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.725361 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" event={"ID":"612d082a-4654-4fd7-b9ce-483cd5444f00","Type":"ContainerStarted","Data":"75e44c8bfa0ba6560fb861dbcc6069115216df8ce0172935a1b1e662cd091e8a"} Jul 15 15:01:46.725585 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.725424 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" event={"ID":"612d082a-4654-4fd7-b9ce-483cd5444f00","Type":"ContainerStarted","Data":"8dbdf36673459935920f40ed9d77b48499c66e69bdd451cdd9eca4ba04fcfeff"} Jul 15 15:01:46.747840 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:46.747787 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" podStartSLOduration=1.7477713160000001 podStartE2EDuration="1.747771316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:01:46.746151728 +0000 UTC m=+1679.452227859" watchObservedRunningTime="2026-07-15 15:01:46.747771316 +0000 UTC m=+1679.453847421" Jul 15 15:01:47.136980 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:47.136941 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:47.142283 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:47.142257 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:47.731250 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:47.731213 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:47.732128 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:47.732098 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:01:48.867594 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:48.867563 2574 patch_prober.go:28] interesting pod/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 15:01:48.867970 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:48.867612 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" podUID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 15:01:50.484434 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.484406 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:01:50.581693 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.581661 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-credential-socket\") pod \"124154bd-e2bf-41c9-ba1a-1a788c82c654\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " Jul 15 15:01:50.581693 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.581695 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-token\") pod \"124154bd-e2bf-41c9-ba1a-1a788c82c654\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " Jul 15 15:01:50.581921 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.581734 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-data\") pod \"124154bd-e2bf-41c9-ba1a-1a788c82c654\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " Jul 15 15:01:50.581921 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.581766 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-workload-certs\") pod \"124154bd-e2bf-41c9-ba1a-1a788c82c654\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " Jul 15 15:01:50.582040 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.581943 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5dn2\" (UniqueName: \"kubernetes.io/projected/124154bd-e2bf-41c9-ba1a-1a788c82c654-kube-api-access-g5dn2\") pod \"124154bd-e2bf-41c9-ba1a-1a788c82c654\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " Jul 15 15:01:50.582040 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.581991 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-envoy\") pod \"124154bd-e2bf-41c9-ba1a-1a788c82c654\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " Jul 15 15:01:50.582040 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.582022 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124154bd-e2bf-41c9-ba1a-1a788c82c654-istiod-ca-cert\") pod \"124154bd-e2bf-41c9-ba1a-1a788c82c654\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " Jul 15 15:01:50.582196 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.582047 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-podinfo\") pod \"124154bd-e2bf-41c9-ba1a-1a788c82c654\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " Jul 15 15:01:50.582196 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.582063 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "124154bd-e2bf-41c9-ba1a-1a788c82c654" (UID: "124154bd-e2bf-41c9-ba1a-1a788c82c654"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:01:50.582196 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.582085 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-workload-socket\") pod \"124154bd-e2bf-41c9-ba1a-1a788c82c654\" (UID: \"124154bd-e2bf-41c9-ba1a-1a788c82c654\") " Jul 15 15:01:50.582404 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.582382 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124154bd-e2bf-41c9-ba1a-1a788c82c654-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "124154bd-e2bf-41c9-ba1a-1a788c82c654" (UID: "124154bd-e2bf-41c9-ba1a-1a788c82c654"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:01:50.582519 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.582462 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/124154bd-e2bf-41c9-ba1a-1a788c82c654-istiod-ca-cert\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:01:50.582519 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.582476 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-credential-socket\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:01:50.582519 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.582502 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "124154bd-e2bf-41c9-ba1a-1a788c82c654" (UID: "124154bd-e2bf-41c9-ba1a-1a788c82c654"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:01:50.582676 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.582602 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "124154bd-e2bf-41c9-ba1a-1a788c82c654" (UID: "124154bd-e2bf-41c9-ba1a-1a788c82c654"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:01:50.582908 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.582841 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-data" (OuterVolumeSpecName: "istio-data") pod "124154bd-e2bf-41c9-ba1a-1a788c82c654" (UID: "124154bd-e2bf-41c9-ba1a-1a788c82c654"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:01:50.584122 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.584094 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-token" (OuterVolumeSpecName: "istio-token") pod "124154bd-e2bf-41c9-ba1a-1a788c82c654" (UID: "124154bd-e2bf-41c9-ba1a-1a788c82c654"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:01:50.584305 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.584285 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "124154bd-e2bf-41c9-ba1a-1a788c82c654" (UID: "124154bd-e2bf-41c9-ba1a-1a788c82c654"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:01:50.584395 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.584302 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124154bd-e2bf-41c9-ba1a-1a788c82c654-kube-api-access-g5dn2" (OuterVolumeSpecName: "kube-api-access-g5dn2") pod "124154bd-e2bf-41c9-ba1a-1a788c82c654" (UID: "124154bd-e2bf-41c9-ba1a-1a788c82c654"). InnerVolumeSpecName "kube-api-access-g5dn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:01:50.584679 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.584654 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "124154bd-e2bf-41c9-ba1a-1a788c82c654" (UID: "124154bd-e2bf-41c9-ba1a-1a788c82c654"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 15:01:50.683556 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.683521 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-envoy\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:01:50.683556 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.683549 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-podinfo\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:01:50.683556 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.683558 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-workload-socket\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:01:50.683556 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.683566 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-token\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:01:50.683812 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.683575 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-istio-data\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:01:50.683812 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.683583 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/124154bd-e2bf-41c9-ba1a-1a788c82c654-workload-certs\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:01:50.683812 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.683591 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5dn2\" (UniqueName: \"kubernetes.io/projected/124154bd-e2bf-41c9-ba1a-1a788c82c654-kube-api-access-g5dn2\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:01:50.747573 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.747535 2574 generic.go:356] "Generic (PLEG): container finished" podID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerID="3579b2f055e4a0ab855788d7c433a517abd156b899a619b5e09f3a3c6cc51802" exitCode=0 Jul 15 15:01:50.747756 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.747593 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" Jul 15 15:01:50.747756 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.747603 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" event={"ID":"124154bd-e2bf-41c9-ba1a-1a788c82c654","Type":"ContainerDied","Data":"3579b2f055e4a0ab855788d7c433a517abd156b899a619b5e09f3a3c6cc51802"} Jul 15 15:01:50.747756 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.747648 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm" event={"ID":"124154bd-e2bf-41c9-ba1a-1a788c82c654","Type":"ContainerDied","Data":"016eaf605331bf9a4f5ad82f3e96a52ba6794e458e720d5ba9f6d8cdee78a8ba"} Jul 15 15:01:50.747756 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.747669 2574 scope.go:117] "RemoveContainer" containerID="3579b2f055e4a0ab855788d7c433a517abd156b899a619b5e09f3a3c6cc51802" Jul 15 15:01:50.756832 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.756812 2574 scope.go:117] "RemoveContainer" containerID="5bcc5bb2ee3c781a447b2d597b33f192969675c1591a2c8cbdf2d0d65545d9df" Jul 15 15:01:50.764920 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.764899 2574 scope.go:117] "RemoveContainer" containerID="3579b2f055e4a0ab855788d7c433a517abd156b899a619b5e09f3a3c6cc51802" Jul 15 15:01:50.765210 ip-10-0-139-83 kubenswrapper[2574]: E0715 15:01:50.765185 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3579b2f055e4a0ab855788d7c433a517abd156b899a619b5e09f3a3c6cc51802\": container with ID starting with 3579b2f055e4a0ab855788d7c433a517abd156b899a619b5e09f3a3c6cc51802 not found: ID does not exist" containerID="3579b2f055e4a0ab855788d7c433a517abd156b899a619b5e09f3a3c6cc51802" Jul 15 15:01:50.765289 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.765221 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3579b2f055e4a0ab855788d7c433a517abd156b899a619b5e09f3a3c6cc51802"} err="failed to get container status \"3579b2f055e4a0ab855788d7c433a517abd156b899a619b5e09f3a3c6cc51802\": rpc error: code = NotFound desc = could not find container \"3579b2f055e4a0ab855788d7c433a517abd156b899a619b5e09f3a3c6cc51802\": container with ID starting with 3579b2f055e4a0ab855788d7c433a517abd156b899a619b5e09f3a3c6cc51802 not found: ID does not exist" Jul 15 15:01:50.765289 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.765245 2574 scope.go:117] "RemoveContainer" containerID="5bcc5bb2ee3c781a447b2d597b33f192969675c1591a2c8cbdf2d0d65545d9df" Jul 15 15:01:50.765544 ip-10-0-139-83 kubenswrapper[2574]: E0715 15:01:50.765525 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bcc5bb2ee3c781a447b2d597b33f192969675c1591a2c8cbdf2d0d65545d9df\": container with ID starting with 5bcc5bb2ee3c781a447b2d597b33f192969675c1591a2c8cbdf2d0d65545d9df not found: ID does not exist" containerID="5bcc5bb2ee3c781a447b2d597b33f192969675c1591a2c8cbdf2d0d65545d9df" Jul 15 15:01:50.765605 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.765549 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bcc5bb2ee3c781a447b2d597b33f192969675c1591a2c8cbdf2d0d65545d9df"} err="failed to get container status \"5bcc5bb2ee3c781a447b2d597b33f192969675c1591a2c8cbdf2d0d65545d9df\": rpc error: code = NotFound desc = could not find container \"5bcc5bb2ee3c781a447b2d597b33f192969675c1591a2c8cbdf2d0d65545d9df\": container with ID starting with 5bcc5bb2ee3c781a447b2d597b33f192969675c1591a2c8cbdf2d0d65545d9df not found: ID does not exist" Jul 15 15:01:50.769815 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.769792 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm"] Jul 15 15:01:50.777986 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:50.777964 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8a5fa064-openshift-default-7f6876fbf7-stjjm"] Jul 15 15:01:51.084197 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:01:51.084161 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="124154bd-e2bf-41c9-ba1a-1a788c82c654" path="/var/lib/kubelet/pods/124154bd-e2bf-41c9-ba1a-1a788c82c654/volumes" Jul 15 15:02:02.774302 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:02.773269 2574 patch_prober.go:28] interesting pod/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:59204->10.133.0.39:15021: read: connection reset by peer" start-of-body= Jul 15 15:02:02.774302 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:02.773333 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" podUID="612d082a-4654-4fd7-b9ce-483cd5444f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:59204->10.133.0.39:15021: read: connection reset by peer" Jul 15 15:02:02.809485 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:02.809456 2574 generic.go:356] "Generic (PLEG): container finished" podID="612d082a-4654-4fd7-b9ce-483cd5444f00" containerID="75e44c8bfa0ba6560fb861dbcc6069115216df8ce0172935a1b1e662cd091e8a" exitCode=0 Jul 15 15:02:02.809601 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:02.809524 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" event={"ID":"612d082a-4654-4fd7-b9ce-483cd5444f00","Type":"ContainerDied","Data":"75e44c8bfa0ba6560fb861dbcc6069115216df8ce0172935a1b1e662cd091e8a"} Jul 15 15:02:02.809982 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:02.809967 2574 scope.go:117] "RemoveContainer" containerID="75e44c8bfa0ba6560fb861dbcc6069115216df8ce0172935a1b1e662cd091e8a" Jul 15 15:02:02.810944 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:02.810918 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:02:02.811006 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:02.810986 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:02:02.811045 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:02.811017 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:02:03.815918 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:03.815870 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" event={"ID":"612d082a-4654-4fd7-b9ce-483cd5444f00","Type":"ContainerStarted","Data":"3a8494c96da60aaefa3745e59c18b77d91875b1f405b548cb97672651395e88f"} Jul 15 15:02:04.137032 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:04.136937 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:02:04.137173 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:04.137149 2574 patch_prober.go:28] interesting pod/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 15 15:02:04.137219 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:04.137199 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" podUID="612d082a-4654-4fd7-b9ce-483cd5444f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 15 15:02:05.137257 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:05.137217 2574 patch_prober.go:28] interesting pod/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 15 15:02:05.137722 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:05.137273 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" podUID="612d082a-4654-4fd7-b9ce-483cd5444f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 15 15:02:06.137051 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:06.137011 2574 patch_prober.go:28] interesting pod/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 15 15:02:06.137239 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:06.137080 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" podUID="612d082a-4654-4fd7-b9ce-483cd5444f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 15 15:02:07.137049 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:07.137015 2574 patch_prober.go:28] interesting pod/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 15 15:02:07.137453 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:07.137072 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" podUID="612d082a-4654-4fd7-b9ce-483cd5444f00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 15 15:02:08.140687 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:08.140658 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:02:08.141162 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:08.141038 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:02:08.141694 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:08.141676 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:02:51.787524 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:51.787485 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7"] Jul 15 15:02:51.788013 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:51.787734 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" podUID="612d082a-4654-4fd7-b9ce-483cd5444f00" containerName="istio-proxy" containerID="cri-o://3a8494c96da60aaefa3745e59c18b77d91875b1f405b548cb97672651395e88f" gracePeriod=30 Jul 15 15:02:53.143269 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:53.143228 2574 patch_prober.go:28] interesting pod/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 15:02:53.143640 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:53.143312 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" podUID="612d082a-4654-4fd7-b9ce-483cd5444f00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 15:02:57.046682 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.046653 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:02:57.073640 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.073605 2574 generic.go:356] "Generic (PLEG): container finished" podID="612d082a-4654-4fd7-b9ce-483cd5444f00" containerID="3a8494c96da60aaefa3745e59c18b77d91875b1f405b548cb97672651395e88f" exitCode=0 Jul 15 15:02:57.073828 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.073651 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" Jul 15 15:02:57.073828 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.073682 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" event={"ID":"612d082a-4654-4fd7-b9ce-483cd5444f00","Type":"ContainerDied","Data":"3a8494c96da60aaefa3745e59c18b77d91875b1f405b548cb97672651395e88f"} Jul 15 15:02:57.073828 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.073794 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7" event={"ID":"612d082a-4654-4fd7-b9ce-483cd5444f00","Type":"ContainerDied","Data":"8dbdf36673459935920f40ed9d77b48499c66e69bdd451cdd9eca4ba04fcfeff"} Jul 15 15:02:57.073828 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.073823 2574 scope.go:117] "RemoveContainer" containerID="3a8494c96da60aaefa3745e59c18b77d91875b1f405b548cb97672651395e88f" Jul 15 15:02:57.089261 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.089157 2574 scope.go:117] "RemoveContainer" containerID="75e44c8bfa0ba6560fb861dbcc6069115216df8ce0172935a1b1e662cd091e8a" Jul 15 15:02:57.103236 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.103208 2574 scope.go:117] "RemoveContainer" containerID="3a8494c96da60aaefa3745e59c18b77d91875b1f405b548cb97672651395e88f" Jul 15 15:02:57.103646 ip-10-0-139-83 kubenswrapper[2574]: E0715 15:02:57.103612 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8494c96da60aaefa3745e59c18b77d91875b1f405b548cb97672651395e88f\": container with ID starting with 3a8494c96da60aaefa3745e59c18b77d91875b1f405b548cb97672651395e88f not found: ID does not exist" containerID="3a8494c96da60aaefa3745e59c18b77d91875b1f405b548cb97672651395e88f" Jul 15 15:02:57.103759 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.103657 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8494c96da60aaefa3745e59c18b77d91875b1f405b548cb97672651395e88f"} err="failed to get container status \"3a8494c96da60aaefa3745e59c18b77d91875b1f405b548cb97672651395e88f\": rpc error: code = NotFound desc = could not find container \"3a8494c96da60aaefa3745e59c18b77d91875b1f405b548cb97672651395e88f\": container with ID starting with 3a8494c96da60aaefa3745e59c18b77d91875b1f405b548cb97672651395e88f not found: ID does not exist" Jul 15 15:02:57.103759 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.103686 2574 scope.go:117] "RemoveContainer" containerID="75e44c8bfa0ba6560fb861dbcc6069115216df8ce0172935a1b1e662cd091e8a" Jul 15 15:02:57.104042 ip-10-0-139-83 kubenswrapper[2574]: E0715 15:02:57.104009 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e44c8bfa0ba6560fb861dbcc6069115216df8ce0172935a1b1e662cd091e8a\": container with ID starting with 75e44c8bfa0ba6560fb861dbcc6069115216df8ce0172935a1b1e662cd091e8a not found: ID does not exist" containerID="75e44c8bfa0ba6560fb861dbcc6069115216df8ce0172935a1b1e662cd091e8a" Jul 15 15:02:57.104156 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.104053 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e44c8bfa0ba6560fb861dbcc6069115216df8ce0172935a1b1e662cd091e8a"} err="failed to get container status \"75e44c8bfa0ba6560fb861dbcc6069115216df8ce0172935a1b1e662cd091e8a\": rpc error: code = NotFound desc = could not find container \"75e44c8bfa0ba6560fb861dbcc6069115216df8ce0172935a1b1e662cd091e8a\": container with ID starting with 75e44c8bfa0ba6560fb861dbcc6069115216df8ce0172935a1b1e662cd091e8a not found: ID does not exist" Jul 15 15:02:57.183255 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183220 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-podinfo\") pod \"612d082a-4654-4fd7-b9ce-483cd5444f00\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " Jul 15 15:02:57.183418 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183269 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612d082a-4654-4fd7-b9ce-483cd5444f00-istiod-ca-cert\") pod \"612d082a-4654-4fd7-b9ce-483cd5444f00\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " Jul 15 15:02:57.183418 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183311 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-token\") pod \"612d082a-4654-4fd7-b9ce-483cd5444f00\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " Jul 15 15:02:57.183418 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183334 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-workload-certs\") pod \"612d082a-4654-4fd7-b9ce-483cd5444f00\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " Jul 15 15:02:57.183418 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183384 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-data\") pod \"612d082a-4654-4fd7-b9ce-483cd5444f00\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " Jul 15 15:02:57.183418 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183410 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7fbh\" (UniqueName: \"kubernetes.io/projected/612d082a-4654-4fd7-b9ce-483cd5444f00-kube-api-access-x7fbh\") pod \"612d082a-4654-4fd7-b9ce-483cd5444f00\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " Jul 15 15:02:57.183657 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183438 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-workload-socket\") pod \"612d082a-4654-4fd7-b9ce-483cd5444f00\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " Jul 15 15:02:57.183657 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183463 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-envoy\") pod \"612d082a-4654-4fd7-b9ce-483cd5444f00\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " Jul 15 15:02:57.183657 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183529 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-credential-socket\") pod \"612d082a-4654-4fd7-b9ce-483cd5444f00\" (UID: \"612d082a-4654-4fd7-b9ce-483cd5444f00\") " Jul 15 15:02:57.183805 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183729 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "612d082a-4654-4fd7-b9ce-483cd5444f00" (UID: "612d082a-4654-4fd7-b9ce-483cd5444f00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:02:57.183805 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183752 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "612d082a-4654-4fd7-b9ce-483cd5444f00" (UID: "612d082a-4654-4fd7-b9ce-483cd5444f00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:02:57.183805 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183792 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612d082a-4654-4fd7-b9ce-483cd5444f00-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "612d082a-4654-4fd7-b9ce-483cd5444f00" (UID: "612d082a-4654-4fd7-b9ce-483cd5444f00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:02:57.183988 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183934 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/612d082a-4654-4fd7-b9ce-483cd5444f00-istiod-ca-cert\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:02:57.183988 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183957 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-workload-certs\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:02:57.183988 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183971 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-workload-socket\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:02:57.184158 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.183980 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "612d082a-4654-4fd7-b9ce-483cd5444f00" (UID: "612d082a-4654-4fd7-b9ce-483cd5444f00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:02:57.184571 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.184541 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-data" (OuterVolumeSpecName: "istio-data") pod "612d082a-4654-4fd7-b9ce-483cd5444f00" (UID: "612d082a-4654-4fd7-b9ce-483cd5444f00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:02:57.185595 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.185568 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-token" (OuterVolumeSpecName: "istio-token") pod "612d082a-4654-4fd7-b9ce-483cd5444f00" (UID: "612d082a-4654-4fd7-b9ce-483cd5444f00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:02:57.185692 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.185593 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "612d082a-4654-4fd7-b9ce-483cd5444f00" (UID: "612d082a-4654-4fd7-b9ce-483cd5444f00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 15:02:57.185692 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.185686 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "612d082a-4654-4fd7-b9ce-483cd5444f00" (UID: "612d082a-4654-4fd7-b9ce-483cd5444f00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:02:57.185993 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.185971 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612d082a-4654-4fd7-b9ce-483cd5444f00-kube-api-access-x7fbh" (OuterVolumeSpecName: "kube-api-access-x7fbh") pod "612d082a-4654-4fd7-b9ce-483cd5444f00" (UID: "612d082a-4654-4fd7-b9ce-483cd5444f00"). InnerVolumeSpecName "kube-api-access-x7fbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:02:57.284788 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.284738 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-data\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:02:57.284788 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.284777 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7fbh\" (UniqueName: \"kubernetes.io/projected/612d082a-4654-4fd7-b9ce-483cd5444f00-kube-api-access-x7fbh\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:02:57.284788 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.284788 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-envoy\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:02:57.284788 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.284796 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/612d082a-4654-4fd7-b9ce-483cd5444f00-credential-socket\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:02:57.284788 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.284805 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-podinfo\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:02:57.285145 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.284814 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/612d082a-4654-4fd7-b9ce-483cd5444f00-istio-token\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:02:57.398244 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.398207 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7"] Jul 15 15:02:57.405676 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:57.405632 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-fc2c3cfa-gw-openshift-default-64d49f8fdc-m8hc7"] Jul 15 15:02:58.589111 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.589079 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2"] Jul 15 15:02:58.589584 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.589526 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="612d082a-4654-4fd7-b9ce-483cd5444f00" containerName="istio-proxy" Jul 15 15:02:58.589584 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.589542 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="612d082a-4654-4fd7-b9ce-483cd5444f00" containerName="istio-proxy" Jul 15 15:02:58.589584 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.589558 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerName="istio-proxy" Jul 15 15:02:58.589584 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.589564 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerName="istio-proxy" Jul 15 15:02:58.589584 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.589571 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerName="istio-proxy" Jul 15 15:02:58.589584 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.589578 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerName="istio-proxy" Jul 15 15:02:58.589584 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.589590 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="612d082a-4654-4fd7-b9ce-483cd5444f00" containerName="istio-proxy" Jul 15 15:02:58.589974 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.589595 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="612d082a-4654-4fd7-b9ce-483cd5444f00" containerName="istio-proxy" Jul 15 15:02:58.589974 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.589665 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerName="istio-proxy" Jul 15 15:02:58.589974 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.589676 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="612d082a-4654-4fd7-b9ce-483cd5444f00" containerName="istio-proxy" Jul 15 15:02:58.589974 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.589686 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="612d082a-4654-4fd7-b9ce-483cd5444f00" containerName="istio-proxy" Jul 15 15:02:58.594216 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.594194 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.596792 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.596771 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a418840f-openshift-default-dockercfg-gdpsp\"" Jul 15 15:02:58.605761 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.605739 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2"] Jul 15 15:02:58.695773 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.695740 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-token\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.695982 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.695787 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-envoy\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.695982 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.695876 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-workload-certs\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.695982 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.695922 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8072c523-bbca-4cdb-9346-81b4ae1df61f-istiod-ca-cert\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.695982 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.695956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-workload-socket\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.695982 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.695976 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-podinfo\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.696217 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.696047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-credential-socket\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.696217 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.696073 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh8hl\" (UniqueName: \"kubernetes.io/projected/8072c523-bbca-4cdb-9346-81b4ae1df61f-kube-api-access-zh8hl\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.696217 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.696106 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-data\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.796848 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.796803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-workload-certs\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.796848 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.796845 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8072c523-bbca-4cdb-9346-81b4ae1df61f-istiod-ca-cert\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.797144 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.796873 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-workload-socket\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.797144 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.796902 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-podinfo\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.797144 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.796987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-credential-socket\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.797144 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.797021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh8hl\" (UniqueName: \"kubernetes.io/projected/8072c523-bbca-4cdb-9346-81b4ae1df61f-kube-api-access-zh8hl\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.797144 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.797052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-data\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.797144 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.797116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-token\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.797439 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.797154 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-envoy\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.797439 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.797268 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-workload-certs\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.797439 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.797316 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-workload-socket\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.797439 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.797352 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-credential-socket\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.797780 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.797755 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-data\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.798071 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.797941 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8072c523-bbca-4cdb-9346-81b4ae1df61f-istiod-ca-cert\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.799536 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.799506 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-podinfo\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.799640 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.799623 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-envoy\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.806760 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.806725 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-token\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.807376 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.807346 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh8hl\" (UniqueName: \"kubernetes.io/projected/8072c523-bbca-4cdb-9346-81b4ae1df61f-kube-api-access-zh8hl\") pod \"e2e-ait-a418840f-openshift-default-75979c776c-929f2\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:58.905265 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:58.905180 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:02:59.047505 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:59.047477 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2"] Jul 15 15:02:59.049450 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:02:59.049403 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8072c523_bbca_4cdb_9346_81b4ae1df61f.slice/crio-9849e695377f2f4f2e5cf467788ba189c8ca56ebdccf8178571a821e98f487b7 WatchSource:0}: Error finding container 9849e695377f2f4f2e5cf467788ba189c8ca56ebdccf8178571a821e98f487b7: Status 404 returned error can't find the container with id 9849e695377f2f4f2e5cf467788ba189c8ca56ebdccf8178571a821e98f487b7 Jul 15 15:02:59.051429 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:59.051394 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:02:59.051508 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:59.051459 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:02:59.051508 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:59.051488 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:02:59.084385 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:59.084353 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="612d082a-4654-4fd7-b9ce-483cd5444f00" path="/var/lib/kubelet/pods/612d082a-4654-4fd7-b9ce-483cd5444f00/volumes" Jul 15 15:02:59.086813 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:02:59.086787 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" event={"ID":"8072c523-bbca-4cdb-9346-81b4ae1df61f","Type":"ContainerStarted","Data":"9849e695377f2f4f2e5cf467788ba189c8ca56ebdccf8178571a821e98f487b7"} Jul 15 15:03:00.093690 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:00.093647 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" event={"ID":"8072c523-bbca-4cdb-9346-81b4ae1df61f","Type":"ContainerStarted","Data":"31e9771084bf58915120c2d688d862a8772e7f24e67965565520572c161ea220"} Jul 15 15:03:00.117739 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:00.117683 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" podStartSLOduration=2.117662761 podStartE2EDuration="2.117662761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:03:00.115360266 +0000 UTC m=+1752.821436369" watchObservedRunningTime="2026-07-15 15:03:00.117662761 +0000 UTC m=+1752.823738868" Jul 15 15:03:00.905585 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:00.905538 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:03:00.910684 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:00.910654 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:03:01.099281 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:01.099247 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:03:01.100377 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:01.100355 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:03:15.171938 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:15.171897 2574 generic.go:356] "Generic (PLEG): container finished" podID="8072c523-bbca-4cdb-9346-81b4ae1df61f" containerID="31e9771084bf58915120c2d688d862a8772e7f24e67965565520572c161ea220" exitCode=0 Jul 15 15:03:15.172371 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:15.171960 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" event={"ID":"8072c523-bbca-4cdb-9346-81b4ae1df61f","Type":"ContainerDied","Data":"31e9771084bf58915120c2d688d862a8772e7f24e67965565520572c161ea220"} Jul 15 15:03:15.172371 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:15.172352 2574 scope.go:117] "RemoveContainer" containerID="31e9771084bf58915120c2d688d862a8772e7f24e67965565520572c161ea220" Jul 15 15:03:15.173337 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:15.173304 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:03:15.173449 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:15.173392 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:03:15.173449 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:15.173440 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:03:16.099325 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:16.099290 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:03:16.178835 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:16.178795 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" event={"ID":"8072c523-bbca-4cdb-9346-81b4ae1df61f","Type":"ContainerStarted","Data":"035fd010effb75ac9392566943808e8ff43e751723840d6c64942d9d82658d1f"} Jul 15 15:03:16.906450 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:16.906409 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:03:16.907629 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:16.907607 2574 patch_prober.go:28] interesting pod/e2e-ait-a418840f-openshift-default-75979c776c-929f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 15 15:03:16.907679 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:16.907657 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" podUID="8072c523-bbca-4cdb-9346-81b4ae1df61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 15 15:03:17.906584 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:17.906549 2574 patch_prober.go:28] interesting pod/e2e-ait-a418840f-openshift-default-75979c776c-929f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 15 15:03:17.906994 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:17.906603 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" podUID="8072c523-bbca-4cdb-9346-81b4ae1df61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 15 15:03:18.906634 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:18.906596 2574 patch_prober.go:28] interesting pod/e2e-ait-a418840f-openshift-default-75979c776c-929f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 15 15:03:18.907059 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:18.906656 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" podUID="8072c523-bbca-4cdb-9346-81b4ae1df61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 15 15:03:19.906002 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:19.905961 2574 patch_prober.go:28] interesting pod/e2e-ait-a418840f-openshift-default-75979c776c-929f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 15 15:03:19.906236 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:19.906019 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" podUID="8072c523-bbca-4cdb-9346-81b4ae1df61f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 15 15:03:20.910047 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:20.910006 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:03:20.910458 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:20.910382 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:03:20.911039 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:20.911019 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:03:48.998110 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:48.998081 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 15:03:49.001087 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:03:49.001066 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 15:04:21.043290 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:21.043249 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2"] Jul 15 15:04:21.043862 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:21.043511 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" podUID="8072c523-bbca-4cdb-9346-81b4ae1df61f" containerName="istio-proxy" containerID="cri-o://035fd010effb75ac9392566943808e8ff43e751723840d6c64942d9d82658d1f" gracePeriod=30 Jul 15 15:04:26.302571 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.302544 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:04:26.389227 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389199 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8072c523-bbca-4cdb-9346-81b4ae1df61f-istiod-ca-cert\") pod \"8072c523-bbca-4cdb-9346-81b4ae1df61f\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " Jul 15 15:04:26.389384 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389256 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-envoy\") pod \"8072c523-bbca-4cdb-9346-81b4ae1df61f\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " Jul 15 15:04:26.389384 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389280 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-credential-socket\") pod \"8072c523-bbca-4cdb-9346-81b4ae1df61f\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " Jul 15 15:04:26.389384 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389323 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-token\") pod \"8072c523-bbca-4cdb-9346-81b4ae1df61f\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " Jul 15 15:04:26.389384 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389351 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-workload-certs\") pod \"8072c523-bbca-4cdb-9346-81b4ae1df61f\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " Jul 15 15:04:26.389599 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389414 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh8hl\" (UniqueName: \"kubernetes.io/projected/8072c523-bbca-4cdb-9346-81b4ae1df61f-kube-api-access-zh8hl\") pod \"8072c523-bbca-4cdb-9346-81b4ae1df61f\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " Jul 15 15:04:26.389599 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389439 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-workload-socket\") pod \"8072c523-bbca-4cdb-9346-81b4ae1df61f\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " Jul 15 15:04:26.389599 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389474 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-podinfo\") pod \"8072c523-bbca-4cdb-9346-81b4ae1df61f\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " Jul 15 15:04:26.389599 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389515 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-data\") pod \"8072c523-bbca-4cdb-9346-81b4ae1df61f\" (UID: \"8072c523-bbca-4cdb-9346-81b4ae1df61f\") " Jul 15 15:04:26.389803 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389681 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8072c523-bbca-4cdb-9346-81b4ae1df61f" (UID: "8072c523-bbca-4cdb-9346-81b4ae1df61f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:04:26.389803 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389699 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8072c523-bbca-4cdb-9346-81b4ae1df61f" (UID: "8072c523-bbca-4cdb-9346-81b4ae1df61f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:04:26.389803 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389709 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8072c523-bbca-4cdb-9346-81b4ae1df61f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8072c523-bbca-4cdb-9346-81b4ae1df61f" (UID: "8072c523-bbca-4cdb-9346-81b4ae1df61f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:04:26.389999 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389818 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8072c523-bbca-4cdb-9346-81b4ae1df61f" (UID: "8072c523-bbca-4cdb-9346-81b4ae1df61f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:04:26.389999 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389955 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-credential-socket\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:04:26.389999 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389978 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-workload-certs\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:04:26.389999 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.389994 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-workload-socket\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:04:26.390207 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.390007 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8072c523-bbca-4cdb-9346-81b4ae1df61f-istiod-ca-cert\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:04:26.390414 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.390391 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-data" (OuterVolumeSpecName: "istio-data") pod "8072c523-bbca-4cdb-9346-81b4ae1df61f" (UID: "8072c523-bbca-4cdb-9346-81b4ae1df61f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:04:26.391523 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.391501 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8072c523-bbca-4cdb-9346-81b4ae1df61f" (UID: "8072c523-bbca-4cdb-9346-81b4ae1df61f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:04:26.391640 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.391593 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8072c523-bbca-4cdb-9346-81b4ae1df61f" (UID: "8072c523-bbca-4cdb-9346-81b4ae1df61f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 15:04:26.391913 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.391895 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-token" (OuterVolumeSpecName: "istio-token") pod "8072c523-bbca-4cdb-9346-81b4ae1df61f" (UID: "8072c523-bbca-4cdb-9346-81b4ae1df61f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:04:26.391991 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.391925 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8072c523-bbca-4cdb-9346-81b4ae1df61f-kube-api-access-zh8hl" (OuterVolumeSpecName: "kube-api-access-zh8hl") pod "8072c523-bbca-4cdb-9346-81b4ae1df61f" (UID: "8072c523-bbca-4cdb-9346-81b4ae1df61f"). InnerVolumeSpecName "kube-api-access-zh8hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:04:26.490435 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.490404 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-envoy\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:04:26.490435 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.490430 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-token\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:04:26.490435 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.490440 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh8hl\" (UniqueName: \"kubernetes.io/projected/8072c523-bbca-4cdb-9346-81b4ae1df61f-kube-api-access-zh8hl\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:04:26.490649 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.490449 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-podinfo\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:04:26.490649 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.490458 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8072c523-bbca-4cdb-9346-81b4ae1df61f-istio-data\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:04:26.508906 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.508852 2574 generic.go:356] "Generic (PLEG): container finished" podID="8072c523-bbca-4cdb-9346-81b4ae1df61f" containerID="035fd010effb75ac9392566943808e8ff43e751723840d6c64942d9d82658d1f" exitCode=0 Jul 15 15:04:26.509013 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.508923 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" Jul 15 15:04:26.509013 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.508933 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" event={"ID":"8072c523-bbca-4cdb-9346-81b4ae1df61f","Type":"ContainerDied","Data":"035fd010effb75ac9392566943808e8ff43e751723840d6c64942d9d82658d1f"} Jul 15 15:04:26.509013 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.508982 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2" event={"ID":"8072c523-bbca-4cdb-9346-81b4ae1df61f","Type":"ContainerDied","Data":"9849e695377f2f4f2e5cf467788ba189c8ca56ebdccf8178571a821e98f487b7"} Jul 15 15:04:26.509013 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.509005 2574 scope.go:117] "RemoveContainer" containerID="035fd010effb75ac9392566943808e8ff43e751723840d6c64942d9d82658d1f" Jul 15 15:04:26.517730 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.517713 2574 scope.go:117] "RemoveContainer" containerID="31e9771084bf58915120c2d688d862a8772e7f24e67965565520572c161ea220" Jul 15 15:04:26.525102 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.525084 2574 scope.go:117] "RemoveContainer" containerID="035fd010effb75ac9392566943808e8ff43e751723840d6c64942d9d82658d1f" Jul 15 15:04:26.525328 ip-10-0-139-83 kubenswrapper[2574]: E0715 15:04:26.525311 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035fd010effb75ac9392566943808e8ff43e751723840d6c64942d9d82658d1f\": container with ID starting with 035fd010effb75ac9392566943808e8ff43e751723840d6c64942d9d82658d1f not found: ID does not exist" containerID="035fd010effb75ac9392566943808e8ff43e751723840d6c64942d9d82658d1f" Jul 15 15:04:26.525388 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.525335 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035fd010effb75ac9392566943808e8ff43e751723840d6c64942d9d82658d1f"} err="failed to get container status \"035fd010effb75ac9392566943808e8ff43e751723840d6c64942d9d82658d1f\": rpc error: code = NotFound desc = could not find container \"035fd010effb75ac9392566943808e8ff43e751723840d6c64942d9d82658d1f\": container with ID starting with 035fd010effb75ac9392566943808e8ff43e751723840d6c64942d9d82658d1f not found: ID does not exist" Jul 15 15:04:26.525388 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.525350 2574 scope.go:117] "RemoveContainer" containerID="31e9771084bf58915120c2d688d862a8772e7f24e67965565520572c161ea220" Jul 15 15:04:26.525547 ip-10-0-139-83 kubenswrapper[2574]: E0715 15:04:26.525528 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e9771084bf58915120c2d688d862a8772e7f24e67965565520572c161ea220\": container with ID starting with 31e9771084bf58915120c2d688d862a8772e7f24e67965565520572c161ea220 not found: ID does not exist" containerID="31e9771084bf58915120c2d688d862a8772e7f24e67965565520572c161ea220" Jul 15 15:04:26.525585 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.525555 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e9771084bf58915120c2d688d862a8772e7f24e67965565520572c161ea220"} err="failed to get container status \"31e9771084bf58915120c2d688d862a8772e7f24e67965565520572c161ea220\": rpc error: code = NotFound desc = could not find container \"31e9771084bf58915120c2d688d862a8772e7f24e67965565520572c161ea220\": container with ID starting with 31e9771084bf58915120c2d688d862a8772e7f24e67965565520572c161ea220 not found: ID does not exist" Jul 15 15:04:26.532359 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.532339 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2"] Jul 15 15:04:26.538827 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:26.538807 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a418840f-openshift-default-75979c776c-929f2"] Jul 15 15:04:27.084109 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:04:27.084080 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8072c523-bbca-4cdb-9346-81b4ae1df61f" path="/var/lib/kubelet/pods/8072c523-bbca-4cdb-9346-81b4ae1df61f/volumes" Jul 15 15:08:49.025988 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:08:49.025961 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 15:08:49.029719 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:08:49.029699 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 15:09:13.417999 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.417963 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv"] Jul 15 15:09:13.418398 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.418362 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8072c523-bbca-4cdb-9346-81b4ae1df61f" containerName="istio-proxy" Jul 15 15:09:13.418398 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.418374 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8072c523-bbca-4cdb-9346-81b4ae1df61f" containerName="istio-proxy" Jul 15 15:09:13.418398 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.418388 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8072c523-bbca-4cdb-9346-81b4ae1df61f" containerName="istio-proxy" Jul 15 15:09:13.418398 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.418396 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8072c523-bbca-4cdb-9346-81b4ae1df61f" containerName="istio-proxy" Jul 15 15:09:13.418527 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.418449 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8072c523-bbca-4cdb-9346-81b4ae1df61f" containerName="istio-proxy" Jul 15 15:09:13.418527 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.418458 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="124154bd-e2bf-41c9-ba1a-1a788c82c654" containerName="istio-proxy" Jul 15 15:09:13.418527 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.418472 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8072c523-bbca-4cdb-9346-81b4ae1df61f" containerName="istio-proxy" Jul 15 15:09:13.421590 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.421574 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv" Jul 15 15:09:13.424051 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.424031 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 15:09:13.424166 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.424079 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kbgk\"" Jul 15 15:09:13.432813 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.432791 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv"] Jul 15 15:09:13.539944 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.539903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c327984-cfe6-46b9-9b35-1bfb416d8a9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv\" (UID: \"8c327984-cfe6-46b9-9b35-1bfb416d8a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv" Jul 15 15:09:13.640505 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.640479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c327984-cfe6-46b9-9b35-1bfb416d8a9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv\" (UID: \"8c327984-cfe6-46b9-9b35-1bfb416d8a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv" Jul 15 15:09:13.641059 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.641041 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c327984-cfe6-46b9-9b35-1bfb416d8a9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv\" (UID: \"8c327984-cfe6-46b9-9b35-1bfb416d8a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv" Jul 15 15:09:13.731731 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.731663 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv" Jul 15 15:09:13.858532 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.858469 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv"] Jul 15 15:09:13.860981 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:09:13.860955 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c327984_cfe6_46b9_9b35_1bfb416d8a9b.slice/crio-0309270ee9cadd4f6d4dad11e141f5c0958eea53838c1a292006188884fbc3c8 WatchSource:0}: Error finding container 0309270ee9cadd4f6d4dad11e141f5c0958eea53838c1a292006188884fbc3c8: Status 404 returned error can't find the container with id 0309270ee9cadd4f6d4dad11e141f5c0958eea53838c1a292006188884fbc3c8 Jul 15 15:09:13.863111 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:13.863095 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 15:09:14.861273 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:14.861237 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv" event={"ID":"8c327984-cfe6-46b9-9b35-1bfb416d8a9b","Type":"ContainerStarted","Data":"0309270ee9cadd4f6d4dad11e141f5c0958eea53838c1a292006188884fbc3c8"} Jul 15 15:09:16.871934 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:16.871903 2574 generic.go:356] "Generic (PLEG): container finished" podID="8c327984-cfe6-46b9-9b35-1bfb416d8a9b" containerID="47e45638f6fd61869824c047204b4a9ed34ef23a584474ca7f1cf98598b96994" exitCode=0 Jul 15 15:09:16.872320 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:16.871991 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv" event={"ID":"8c327984-cfe6-46b9-9b35-1bfb416d8a9b","Type":"ContainerDied","Data":"47e45638f6fd61869824c047204b4a9ed34ef23a584474ca7f1cf98598b96994"} Jul 15 15:09:18.005401 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:18.005379 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv" Jul 15 15:09:18.183139 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:18.183057 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c327984-cfe6-46b9-9b35-1bfb416d8a9b-maas-api-service-ca\") pod \"8c327984-cfe6-46b9-9b35-1bfb416d8a9b\" (UID: \"8c327984-cfe6-46b9-9b35-1bfb416d8a9b\") " Jul 15 15:09:18.183386 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:18.183366 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c327984-cfe6-46b9-9b35-1bfb416d8a9b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c327984-cfe6-46b9-9b35-1bfb416d8a9b" (UID: "8c327984-cfe6-46b9-9b35-1bfb416d8a9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:09:18.283724 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:18.283697 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c327984-cfe6-46b9-9b35-1bfb416d8a9b-maas-api-service-ca\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:09:18.882367 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:18.882337 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv" Jul 15 15:09:18.882541 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:18.882372 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv" event={"ID":"8c327984-cfe6-46b9-9b35-1bfb416d8a9b","Type":"ContainerDied","Data":"0309270ee9cadd4f6d4dad11e141f5c0958eea53838c1a292006188884fbc3c8"} Jul 15 15:09:18.882541 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:09:18.882420 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0309270ee9cadd4f6d4dad11e141f5c0958eea53838c1a292006188884fbc3c8" Jul 15 15:11:14.914099 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.914021 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd"] Jul 15 15:11:14.914522 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.914415 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c327984-cfe6-46b9-9b35-1bfb416d8a9b" containerName="revoke-keys" Jul 15 15:11:14.914522 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.914427 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c327984-cfe6-46b9-9b35-1bfb416d8a9b" containerName="revoke-keys" Jul 15 15:11:14.914522 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.914489 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c327984-cfe6-46b9-9b35-1bfb416d8a9b" containerName="revoke-keys" Jul 15 15:11:14.916572 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.916556 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:14.919096 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.919074 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-1e3693-openshift-default-dockercfg-lbk5p\"" Jul 15 15:11:14.927771 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.927747 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd"] Jul 15 15:11:14.967512 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.967479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-envoy\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:14.967649 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.967517 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-workload-certs\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:14.967649 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.967537 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq5mt\" (UniqueName: \"kubernetes.io/projected/6feda70c-9e55-486a-b6e8-d79c7e34229d-kube-api-access-wq5mt\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:14.967649 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.967564 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-podinfo\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:14.967649 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.967597 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-token\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:14.967649 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.967629 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-data\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:14.967649 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.967643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6feda70c-9e55-486a-b6e8-d79c7e34229d-istiod-ca-cert\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:14.967876 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.967658 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-workload-socket\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:14.967876 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:14.967674 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-credential-socket\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.068448 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.068411 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-data\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.068448 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.068446 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6feda70c-9e55-486a-b6e8-d79c7e34229d-istiod-ca-cert\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.068698 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.068465 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-workload-socket\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.068698 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.068482 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-credential-socket\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.068698 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.068521 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-envoy\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.068698 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.068544 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-workload-certs\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.068698 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.068656 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq5mt\" (UniqueName: \"kubernetes.io/projected/6feda70c-9e55-486a-b6e8-d79c7e34229d-kube-api-access-wq5mt\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.068698 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.068698 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-podinfo\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.069042 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.068737 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-token\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.069042 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.068879 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-workload-socket\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.069042 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.068925 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-credential-socket\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.069195 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.069055 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-workload-certs\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.069258 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.069227 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-data\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.069339 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.069321 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6feda70c-9e55-486a-b6e8-d79c7e34229d-istiod-ca-cert\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.070927 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.070907 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-envoy\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.071136 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.071119 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-podinfo\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.077274 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.077251 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-token\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.077353 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.077314 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq5mt\" (UniqueName: \"kubernetes.io/projected/6feda70c-9e55-486a-b6e8-d79c7e34229d-kube-api-access-wq5mt\") pod \"e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.226771 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.226684 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:15.360251 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.360222 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd"] Jul 15 15:11:15.362153 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:11:15.362125 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6feda70c_9e55_486a_b6e8_d79c7e34229d.slice/crio-28b821120ce9973672eb1e175ba5dd49310b6556b432706dc3bc928b7b860693 WatchSource:0}: Error finding container 28b821120ce9973672eb1e175ba5dd49310b6556b432706dc3bc928b7b860693: Status 404 returned error can't find the container with id 28b821120ce9973672eb1e175ba5dd49310b6556b432706dc3bc928b7b860693 Jul 15 15:11:15.364207 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.364171 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:11:15.364296 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.364241 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:11:15.364335 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.364302 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:11:15.443615 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.443581 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" event={"ID":"6feda70c-9e55-486a-b6e8-d79c7e34229d","Type":"ContainerStarted","Data":"ef5f6e099c74aaa76ecd51aee88be5e5408be90eb156da4443cab6fe32e8d3d6"} Jul 15 15:11:15.443760 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.443643 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" event={"ID":"6feda70c-9e55-486a-b6e8-d79c7e34229d","Type":"ContainerStarted","Data":"28b821120ce9973672eb1e175ba5dd49310b6556b432706dc3bc928b7b860693"} Jul 15 15:11:15.465400 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:15.465344 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" podStartSLOduration=1.465323583 podStartE2EDuration="1.465323583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:11:15.46359248 +0000 UTC m=+2248.169668587" watchObservedRunningTime="2026-07-15 15:11:15.465323583 +0000 UTC m=+2248.171399688" Jul 15 15:11:16.227551 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:16.227524 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:16.232306 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:16.232282 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:16.448744 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:16.448715 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:16.449740 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:16.449716 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:32.490039 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:32.489999 2574 patch_prober.go:28] interesting pod/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 15:11:32.490443 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:32.490063 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" podUID="6feda70c-9e55-486a-b6e8-d79c7e34229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 15:11:33.531010 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:33.530937 2574 generic.go:356] "Generic (PLEG): container finished" podID="6feda70c-9e55-486a-b6e8-d79c7e34229d" containerID="ef5f6e099c74aaa76ecd51aee88be5e5408be90eb156da4443cab6fe32e8d3d6" exitCode=0 Jul 15 15:11:33.531379 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:33.531009 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" event={"ID":"6feda70c-9e55-486a-b6e8-d79c7e34229d","Type":"ContainerDied","Data":"ef5f6e099c74aaa76ecd51aee88be5e5408be90eb156da4443cab6fe32e8d3d6"} Jul 15 15:11:33.531444 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:33.531430 2574 scope.go:117] "RemoveContainer" containerID="ef5f6e099c74aaa76ecd51aee88be5e5408be90eb156da4443cab6fe32e8d3d6" Jul 15 15:11:33.532539 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:33.532512 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:11:33.532612 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:33.532583 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:11:33.532649 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:33.532614 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 15 15:11:34.537978 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:34.537946 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" event={"ID":"6feda70c-9e55-486a-b6e8-d79c7e34229d","Type":"ContainerStarted","Data":"5401bc88aca2adfc9ac60b1f52d5899a8f691a7e81a4e1706d348d3cab6039cd"} Jul 15 15:11:35.227297 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:35.227264 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:35.227584 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:35.227561 2574 patch_prober.go:28] interesting pod/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 15 15:11:35.227682 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:35.227614 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" podUID="6feda70c-9e55-486a-b6e8-d79c7e34229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 15 15:11:36.227597 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:36.227562 2574 patch_prober.go:28] interesting pod/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 15 15:11:36.228092 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:36.227630 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" podUID="6feda70c-9e55-486a-b6e8-d79c7e34229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 15 15:11:37.227748 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:37.227717 2574 patch_prober.go:28] interesting pod/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 15 15:11:37.228241 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:37.227778 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" podUID="6feda70c-9e55-486a-b6e8-d79c7e34229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 15 15:11:38.227311 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:38.227274 2574 patch_prober.go:28] interesting pod/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 15 15:11:38.227492 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:38.227332 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" podUID="6feda70c-9e55-486a-b6e8-d79c7e34229d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 15 15:11:39.230547 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:39.230519 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:39.230980 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:39.230780 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:11:39.231388 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:11:39.231371 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:12:31.806292 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:31.806252 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b"] Jul 15 15:12:31.808815 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:31.808795 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b" Jul 15 15:12:31.811406 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:31.811372 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 15:12:31.811406 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:31.811385 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kbgk\"" Jul 15 15:12:31.817253 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:31.817226 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b"] Jul 15 15:12:31.878600 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:31.878566 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28aeefae-c8bd-469b-9dc0-cf4ab17f50ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b\" (UID: \"28aeefae-c8bd-469b-9dc0-cf4ab17f50ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b" Jul 15 15:12:31.979852 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:31.979814 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28aeefae-c8bd-469b-9dc0-cf4ab17f50ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b\" (UID: \"28aeefae-c8bd-469b-9dc0-cf4ab17f50ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b" Jul 15 15:12:31.980479 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:31.980460 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28aeefae-c8bd-469b-9dc0-cf4ab17f50ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b\" (UID: \"28aeefae-c8bd-469b-9dc0-cf4ab17f50ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b" Jul 15 15:12:32.121059 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:32.120974 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b" Jul 15 15:12:32.254912 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:32.254871 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b"] Jul 15 15:12:32.257299 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:12:32.257263 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28aeefae_c8bd_469b_9dc0_cf4ab17f50ae.slice/crio-53128f0cf4283a5c716b261788b62e1eb45b04ca52eee2e449bc98752bceabd0 WatchSource:0}: Error finding container 53128f0cf4283a5c716b261788b62e1eb45b04ca52eee2e449bc98752bceabd0: Status 404 returned error can't find the container with id 53128f0cf4283a5c716b261788b62e1eb45b04ca52eee2e449bc98752bceabd0 Jul 15 15:12:32.821098 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:32.821060 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b" event={"ID":"28aeefae-c8bd-469b-9dc0-cf4ab17f50ae","Type":"ContainerStarted","Data":"17fcb8b5416d63f9fd9540c175c25cd3d34203c5a3edeaafe5de02c765312c09"} Jul 15 15:12:32.821510 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:32.821114 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b" event={"ID":"28aeefae-c8bd-469b-9dc0-cf4ab17f50ae","Type":"ContainerStarted","Data":"53128f0cf4283a5c716b261788b62e1eb45b04ca52eee2e449bc98752bceabd0"} Jul 15 15:12:32.837857 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:32.837813 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b" podStartSLOduration=1.8377976569999999 podStartE2EDuration="1.837797657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:12:32.836943674 +0000 UTC m=+2325.543019780" watchObservedRunningTime="2026-07-15 15:12:32.837797657 +0000 UTC m=+2325.543873760" Jul 15 15:12:33.828290 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:33.828259 2574 generic.go:356] "Generic (PLEG): container finished" podID="28aeefae-c8bd-469b-9dc0-cf4ab17f50ae" containerID="17fcb8b5416d63f9fd9540c175c25cd3d34203c5a3edeaafe5de02c765312c09" exitCode=0 Jul 15 15:12:33.828685 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:33.828346 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b" event={"ID":"28aeefae-c8bd-469b-9dc0-cf4ab17f50ae","Type":"ContainerDied","Data":"17fcb8b5416d63f9fd9540c175c25cd3d34203c5a3edeaafe5de02c765312c09"} Jul 15 15:12:34.966827 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:34.966801 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b" Jul 15 15:12:35.002302 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:35.002276 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28aeefae-c8bd-469b-9dc0-cf4ab17f50ae-maas-api-service-ca\") pod \"28aeefae-c8bd-469b-9dc0-cf4ab17f50ae\" (UID: \"28aeefae-c8bd-469b-9dc0-cf4ab17f50ae\") " Jul 15 15:12:35.002617 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:35.002592 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28aeefae-c8bd-469b-9dc0-cf4ab17f50ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "28aeefae-c8bd-469b-9dc0-cf4ab17f50ae" (UID: "28aeefae-c8bd-469b-9dc0-cf4ab17f50ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:12:35.102711 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:35.102687 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28aeefae-c8bd-469b-9dc0-cf4ab17f50ae-maas-api-service-ca\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:12:35.838997 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:35.838965 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b" Jul 15 15:12:35.838997 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:35.838978 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b" event={"ID":"28aeefae-c8bd-469b-9dc0-cf4ab17f50ae","Type":"ContainerDied","Data":"53128f0cf4283a5c716b261788b62e1eb45b04ca52eee2e449bc98752bceabd0"} Jul 15 15:12:35.839221 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:12:35.839023 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53128f0cf4283a5c716b261788b62e1eb45b04ca52eee2e449bc98752bceabd0" Jul 15 15:13:23.751553 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:23.751514 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd"] Jul 15 15:13:23.752265 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:23.751751 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" podUID="6feda70c-9e55-486a-b6e8-d79c7e34229d" containerName="istio-proxy" containerID="cri-o://5401bc88aca2adfc9ac60b1f52d5899a8f691a7e81a4e1706d348d3cab6039cd" gracePeriod=30 Jul 15 15:13:24.233088 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:24.233051 2574 patch_prober.go:28] interesting pod/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 15:13:24.233275 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:24.233151 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" podUID="6feda70c-9e55-486a-b6e8-d79c7e34229d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 15:13:28.999364 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:28.999340 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:13:29.094369 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.094332 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-credential-socket\") pod \"6feda70c-9e55-486a-b6e8-d79c7e34229d\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " Jul 15 15:13:29.094554 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.094378 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-workload-certs\") pod \"6feda70c-9e55-486a-b6e8-d79c7e34229d\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " Jul 15 15:13:29.094554 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.094404 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-envoy\") pod \"6feda70c-9e55-486a-b6e8-d79c7e34229d\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " Jul 15 15:13:29.094554 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.094430 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-token\") pod \"6feda70c-9e55-486a-b6e8-d79c7e34229d\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " Jul 15 15:13:29.094554 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.094455 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6feda70c-9e55-486a-b6e8-d79c7e34229d-istiod-ca-cert\") pod \"6feda70c-9e55-486a-b6e8-d79c7e34229d\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " Jul 15 15:13:29.094554 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.094485 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-data\") pod \"6feda70c-9e55-486a-b6e8-d79c7e34229d\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " Jul 15 15:13:29.094554 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.094518 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-workload-socket\") pod \"6feda70c-9e55-486a-b6e8-d79c7e34229d\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " Jul 15 15:13:29.094859 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.094562 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq5mt\" (UniqueName: \"kubernetes.io/projected/6feda70c-9e55-486a-b6e8-d79c7e34229d-kube-api-access-wq5mt\") pod \"6feda70c-9e55-486a-b6e8-d79c7e34229d\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " Jul 15 15:13:29.094859 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.094610 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-podinfo\") pod \"6feda70c-9e55-486a-b6e8-d79c7e34229d\" (UID: \"6feda70c-9e55-486a-b6e8-d79c7e34229d\") " Jul 15 15:13:29.094859 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.094610 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6feda70c-9e55-486a-b6e8-d79c7e34229d" (UID: "6feda70c-9e55-486a-b6e8-d79c7e34229d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:13:29.094859 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.094703 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6feda70c-9e55-486a-b6e8-d79c7e34229d" (UID: "6feda70c-9e55-486a-b6e8-d79c7e34229d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:13:29.095128 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.094927 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6feda70c-9e55-486a-b6e8-d79c7e34229d" (UID: "6feda70c-9e55-486a-b6e8-d79c7e34229d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:13:29.095128 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.094992 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-credential-socket\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:13:29.095128 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.095008 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-workload-certs\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:13:29.095128 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.095019 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-workload-socket\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:13:29.095454 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.095421 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-data" (OuterVolumeSpecName: "istio-data") pod "6feda70c-9e55-486a-b6e8-d79c7e34229d" (UID: "6feda70c-9e55-486a-b6e8-d79c7e34229d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:13:29.095591 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.095493 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6feda70c-9e55-486a-b6e8-d79c7e34229d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6feda70c-9e55-486a-b6e8-d79c7e34229d" (UID: "6feda70c-9e55-486a-b6e8-d79c7e34229d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:13:29.096800 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.096770 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6feda70c-9e55-486a-b6e8-d79c7e34229d" (UID: "6feda70c-9e55-486a-b6e8-d79c7e34229d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:13:29.096800 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.096780 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6feda70c-9e55-486a-b6e8-d79c7e34229d" (UID: "6feda70c-9e55-486a-b6e8-d79c7e34229d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 15:13:29.096800 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.096781 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-token" (OuterVolumeSpecName: "istio-token") pod "6feda70c-9e55-486a-b6e8-d79c7e34229d" (UID: "6feda70c-9e55-486a-b6e8-d79c7e34229d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:13:29.096986 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.096838 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6feda70c-9e55-486a-b6e8-d79c7e34229d-kube-api-access-wq5mt" (OuterVolumeSpecName: "kube-api-access-wq5mt") pod "6feda70c-9e55-486a-b6e8-d79c7e34229d" (UID: "6feda70c-9e55-486a-b6e8-d79c7e34229d"). InnerVolumeSpecName "kube-api-access-wq5mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:13:29.106691 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.106669 2574 generic.go:356] "Generic (PLEG): container finished" podID="6feda70c-9e55-486a-b6e8-d79c7e34229d" containerID="5401bc88aca2adfc9ac60b1f52d5899a8f691a7e81a4e1706d348d3cab6039cd" exitCode=0 Jul 15 15:13:29.106783 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.106719 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" Jul 15 15:13:29.106783 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.106748 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" event={"ID":"6feda70c-9e55-486a-b6e8-d79c7e34229d","Type":"ContainerDied","Data":"5401bc88aca2adfc9ac60b1f52d5899a8f691a7e81a4e1706d348d3cab6039cd"} Jul 15 15:13:29.106872 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.106803 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd" event={"ID":"6feda70c-9e55-486a-b6e8-d79c7e34229d","Type":"ContainerDied","Data":"28b821120ce9973672eb1e175ba5dd49310b6556b432706dc3bc928b7b860693"} Jul 15 15:13:29.106872 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.106835 2574 scope.go:117] "RemoveContainer" containerID="5401bc88aca2adfc9ac60b1f52d5899a8f691a7e81a4e1706d348d3cab6039cd" Jul 15 15:13:29.115758 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.115741 2574 scope.go:117] "RemoveContainer" containerID="ef5f6e099c74aaa76ecd51aee88be5e5408be90eb156da4443cab6fe32e8d3d6" Jul 15 15:13:29.123278 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.123261 2574 scope.go:117] "RemoveContainer" containerID="5401bc88aca2adfc9ac60b1f52d5899a8f691a7e81a4e1706d348d3cab6039cd" Jul 15 15:13:29.123519 ip-10-0-139-83 kubenswrapper[2574]: E0715 15:13:29.123501 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5401bc88aca2adfc9ac60b1f52d5899a8f691a7e81a4e1706d348d3cab6039cd\": container with ID starting with 5401bc88aca2adfc9ac60b1f52d5899a8f691a7e81a4e1706d348d3cab6039cd not found: ID does not exist" containerID="5401bc88aca2adfc9ac60b1f52d5899a8f691a7e81a4e1706d348d3cab6039cd" Jul 15 15:13:29.123586 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.123525 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5401bc88aca2adfc9ac60b1f52d5899a8f691a7e81a4e1706d348d3cab6039cd"} err="failed to get container status \"5401bc88aca2adfc9ac60b1f52d5899a8f691a7e81a4e1706d348d3cab6039cd\": rpc error: code = NotFound desc = could not find container \"5401bc88aca2adfc9ac60b1f52d5899a8f691a7e81a4e1706d348d3cab6039cd\": container with ID starting with 5401bc88aca2adfc9ac60b1f52d5899a8f691a7e81a4e1706d348d3cab6039cd not found: ID does not exist" Jul 15 15:13:29.123586 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.123547 2574 scope.go:117] "RemoveContainer" containerID="ef5f6e099c74aaa76ecd51aee88be5e5408be90eb156da4443cab6fe32e8d3d6" Jul 15 15:13:29.123771 ip-10-0-139-83 kubenswrapper[2574]: E0715 15:13:29.123755 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef5f6e099c74aaa76ecd51aee88be5e5408be90eb156da4443cab6fe32e8d3d6\": container with ID starting with ef5f6e099c74aaa76ecd51aee88be5e5408be90eb156da4443cab6fe32e8d3d6 not found: ID does not exist" containerID="ef5f6e099c74aaa76ecd51aee88be5e5408be90eb156da4443cab6fe32e8d3d6" Jul 15 15:13:29.123816 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.123774 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef5f6e099c74aaa76ecd51aee88be5e5408be90eb156da4443cab6fe32e8d3d6"} err="failed to get container status \"ef5f6e099c74aaa76ecd51aee88be5e5408be90eb156da4443cab6fe32e8d3d6\": rpc error: code = NotFound desc = could not find container \"ef5f6e099c74aaa76ecd51aee88be5e5408be90eb156da4443cab6fe32e8d3d6\": container with ID starting with ef5f6e099c74aaa76ecd51aee88be5e5408be90eb156da4443cab6fe32e8d3d6 not found: ID does not exist" Jul 15 15:13:29.135544 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.135524 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd"] Jul 15 15:13:29.140832 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.140808 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-1e3693-openshift-default-5b99b4fc77-sq7kd"] Jul 15 15:13:29.196316 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.196277 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-envoy\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:13:29.196316 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.196312 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-token\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:13:29.196316 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.196324 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6feda70c-9e55-486a-b6e8-d79c7e34229d-istiod-ca-cert\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:13:29.196622 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.196335 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-data\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:13:29.196622 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.196346 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq5mt\" (UniqueName: \"kubernetes.io/projected/6feda70c-9e55-486a-b6e8-d79c7e34229d-kube-api-access-wq5mt\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:13:29.196622 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:29.196365 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6feda70c-9e55-486a-b6e8-d79c7e34229d-istio-podinfo\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:13:31.084081 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:31.084045 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6feda70c-9e55-486a-b6e8-d79c7e34229d" path="/var/lib/kubelet/pods/6feda70c-9e55-486a-b6e8-d79c7e34229d/volumes" Jul 15 15:13:48.103161 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.103125 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9"] Jul 15 15:13:48.103541 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.103463 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28aeefae-c8bd-469b-9dc0-cf4ab17f50ae" containerName="revoke-keys" Jul 15 15:13:48.103541 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.103475 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="28aeefae-c8bd-469b-9dc0-cf4ab17f50ae" containerName="revoke-keys" Jul 15 15:13:48.103541 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.103493 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6feda70c-9e55-486a-b6e8-d79c7e34229d" containerName="istio-proxy" Jul 15 15:13:48.103541 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.103498 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6feda70c-9e55-486a-b6e8-d79c7e34229d" containerName="istio-proxy" Jul 15 15:13:48.103541 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.103508 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6feda70c-9e55-486a-b6e8-d79c7e34229d" containerName="istio-proxy" Jul 15 15:13:48.103541 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.103516 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6feda70c-9e55-486a-b6e8-d79c7e34229d" containerName="istio-proxy" Jul 15 15:13:48.103751 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.103577 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="28aeefae-c8bd-469b-9dc0-cf4ab17f50ae" containerName="revoke-keys" Jul 15 15:13:48.103751 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.103585 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6feda70c-9e55-486a-b6e8-d79c7e34229d" containerName="istio-proxy" Jul 15 15:13:48.103751 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.103591 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6feda70c-9e55-486a-b6e8-d79c7e34229d" containerName="istio-proxy" Jul 15 15:13:48.106726 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.106709 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9" Jul 15 15:13:48.109157 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.109135 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kbgk\"" Jul 15 15:13:48.109258 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.109136 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 15:13:48.113679 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.113654 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9"] Jul 15 15:13:48.164633 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.164598 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9\" (UID: \"3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9" Jul 15 15:13:48.265250 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.265210 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9\" (UID: \"3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9" Jul 15 15:13:48.265879 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.265859 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9\" (UID: \"3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9" Jul 15 15:13:48.418938 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.418833 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9" Jul 15 15:13:48.549428 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:48.549404 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9"] Jul 15 15:13:48.551616 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:13:48.551583 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aac4b81_0ac8_4a9d_8725_e8715dd4ba2e.slice/crio-80d18576f9e3ce96dade9ff557bfc5b87e6538b9a4d3eb1e210a6433c90af84c WatchSource:0}: Error finding container 80d18576f9e3ce96dade9ff557bfc5b87e6538b9a4d3eb1e210a6433c90af84c: Status 404 returned error can't find the container with id 80d18576f9e3ce96dade9ff557bfc5b87e6538b9a4d3eb1e210a6433c90af84c Jul 15 15:13:49.056363 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:49.056333 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 15:13:49.061167 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:49.061141 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 15:13:49.205610 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:49.205577 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9" event={"ID":"3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e","Type":"ContainerStarted","Data":"c076e8f2bcfb91dcdf834a232609cc7501b6e8372859507b81e7f31c681d84c0"} Jul 15 15:13:49.206012 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:49.205630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9" event={"ID":"3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e","Type":"ContainerStarted","Data":"80d18576f9e3ce96dade9ff557bfc5b87e6538b9a4d3eb1e210a6433c90af84c"} Jul 15 15:13:49.222808 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:49.222763 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9" podStartSLOduration=1.222747851 podStartE2EDuration="1.222747851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:13:49.221190501 +0000 UTC m=+2401.927266605" watchObservedRunningTime="2026-07-15 15:13:49.222747851 +0000 UTC m=+2401.928823984" Jul 15 15:13:50.213274 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:50.213238 2574 generic.go:356] "Generic (PLEG): container finished" podID="3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e" containerID="c076e8f2bcfb91dcdf834a232609cc7501b6e8372859507b81e7f31c681d84c0" exitCode=0 Jul 15 15:13:50.213735 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:50.213304 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9" event={"ID":"3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e","Type":"ContainerDied","Data":"c076e8f2bcfb91dcdf834a232609cc7501b6e8372859507b81e7f31c681d84c0"} Jul 15 15:13:51.360778 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:51.360754 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9" Jul 15 15:13:51.497519 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:51.497429 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e-maas-api-service-ca\") pod \"3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e\" (UID: \"3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e\") " Jul 15 15:13:51.497789 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:51.497767 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e" (UID: "3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:13:51.598071 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:51.598037 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e-maas-api-service-ca\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:13:52.222946 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:52.222908 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9" event={"ID":"3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e","Type":"ContainerDied","Data":"80d18576f9e3ce96dade9ff557bfc5b87e6538b9a4d3eb1e210a6433c90af84c"} Jul 15 15:13:52.222946 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:52.222925 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9" Jul 15 15:13:52.223206 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:13:52.222958 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80d18576f9e3ce96dade9ff557bfc5b87e6538b9a4d3eb1e210a6433c90af84c" Jul 15 15:14:14.259532 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:14.259454 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-345426-qjk7j"] Jul 15 15:14:14.259950 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:14.259851 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e" containerName="revoke-keys" Jul 15 15:14:14.259950 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:14.259863 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e" containerName="revoke-keys" Jul 15 15:14:14.259950 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:14.259942 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e" containerName="revoke-keys" Jul 15 15:14:14.261919 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:14.261903 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-345426-qjk7j" Jul 15 15:14:14.264495 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:14.264475 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 15:14:14.264623 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:14.264570 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kbgk\"" Jul 15 15:14:14.270933 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:14.270911 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-345426-qjk7j"] Jul 15 15:14:14.292193 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:14.292170 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/221f7ea2-c731-4112-82b5-1f2b6e6f64d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-345426-qjk7j\" (UID: \"221f7ea2-c731-4112-82b5-1f2b6e6f64d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-345426-qjk7j" Jul 15 15:14:14.393510 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:14.393485 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/221f7ea2-c731-4112-82b5-1f2b6e6f64d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-345426-qjk7j\" (UID: \"221f7ea2-c731-4112-82b5-1f2b6e6f64d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-345426-qjk7j" Jul 15 15:14:14.394112 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:14.394095 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/221f7ea2-c731-4112-82b5-1f2b6e6f64d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-345426-qjk7j\" (UID: \"221f7ea2-c731-4112-82b5-1f2b6e6f64d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-345426-qjk7j" Jul 15 15:14:14.572714 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:14.572687 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-345426-qjk7j" Jul 15 15:14:14.701241 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:14.701219 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-345426-qjk7j"] Jul 15 15:14:14.703463 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:14:14.703437 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221f7ea2_c731_4112_82b5_1f2b6e6f64d3.slice/crio-304b729e7e7d6db483306cf53daeb4bec8b859272c580f0e36e366b7ad38e815 WatchSource:0}: Error finding container 304b729e7e7d6db483306cf53daeb4bec8b859272c580f0e36e366b7ad38e815: Status 404 returned error can't find the container with id 304b729e7e7d6db483306cf53daeb4bec8b859272c580f0e36e366b7ad38e815 Jul 15 15:14:15.331972 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:15.331936 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-345426-qjk7j" event={"ID":"221f7ea2-c731-4112-82b5-1f2b6e6f64d3","Type":"ContainerStarted","Data":"9e9e238c081ee6c515fdf88dd7f1df3edd357b359c305a396dca383ace3b8ffa"} Jul 15 15:14:15.332338 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:15.331992 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-345426-qjk7j" event={"ID":"221f7ea2-c731-4112-82b5-1f2b6e6f64d3","Type":"ContainerStarted","Data":"304b729e7e7d6db483306cf53daeb4bec8b859272c580f0e36e366b7ad38e815"} Jul 15 15:14:15.350262 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:15.350212 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-345426-qjk7j" podStartSLOduration=1.3501937480000001 podStartE2EDuration="1.350193748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:14:15.349374492 +0000 UTC m=+2428.055450599" watchObservedRunningTime="2026-07-15 15:14:15.350193748 +0000 UTC m=+2428.056269853" Jul 15 15:14:16.344693 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:16.344660 2574 generic.go:356] "Generic (PLEG): container finished" podID="221f7ea2-c731-4112-82b5-1f2b6e6f64d3" containerID="9e9e238c081ee6c515fdf88dd7f1df3edd357b359c305a396dca383ace3b8ffa" exitCode=0 Jul 15 15:14:16.344693 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:16.344699 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-345426-qjk7j" event={"ID":"221f7ea2-c731-4112-82b5-1f2b6e6f64d3","Type":"ContainerDied","Data":"9e9e238c081ee6c515fdf88dd7f1df3edd357b359c305a396dca383ace3b8ffa"} Jul 15 15:14:17.477731 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:17.477707 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-345426-qjk7j" Jul 15 15:14:17.519812 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:17.519785 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/221f7ea2-c731-4112-82b5-1f2b6e6f64d3-maas-api-service-ca\") pod \"221f7ea2-c731-4112-82b5-1f2b6e6f64d3\" (UID: \"221f7ea2-c731-4112-82b5-1f2b6e6f64d3\") " Jul 15 15:14:17.520137 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:17.520113 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221f7ea2-c731-4112-82b5-1f2b6e6f64d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "221f7ea2-c731-4112-82b5-1f2b6e6f64d3" (UID: "221f7ea2-c731-4112-82b5-1f2b6e6f64d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:14:17.621408 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:17.621347 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/221f7ea2-c731-4112-82b5-1f2b6e6f64d3-maas-api-service-ca\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:14:18.355938 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:18.355876 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-345426-qjk7j" event={"ID":"221f7ea2-c731-4112-82b5-1f2b6e6f64d3","Type":"ContainerDied","Data":"304b729e7e7d6db483306cf53daeb4bec8b859272c580f0e36e366b7ad38e815"} Jul 15 15:14:18.355938 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:18.355924 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-345426-qjk7j" Jul 15 15:14:18.356138 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:14:18.355960 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="304b729e7e7d6db483306cf53daeb4bec8b859272c580f0e36e366b7ad38e815" Jul 15 15:15:00.131130 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:00.131097 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0c81d7-29735475-nbppm"] Jul 15 15:15:00.131589 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:00.131466 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221f7ea2-c731-4112-82b5-1f2b6e6f64d3" containerName="revoke-keys" Jul 15 15:15:00.131589 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:00.131477 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="221f7ea2-c731-4112-82b5-1f2b6e6f64d3" containerName="revoke-keys" Jul 15 15:15:00.131589 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:00.131530 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="221f7ea2-c731-4112-82b5-1f2b6e6f64d3" containerName="revoke-keys" Jul 15 15:15:00.133951 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:00.133932 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0c81d7-29735475-nbppm" Jul 15 15:15:00.136382 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:00.136364 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kbgk\"" Jul 15 15:15:00.143988 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:00.143965 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0c81d7-29735475-nbppm" Jul 15 15:15:00.144853 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:00.144833 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0c81d7-29735475-nbppm"] Jul 15 15:15:00.282711 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:00.282687 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0c81d7-29735475-nbppm"] Jul 15 15:15:00.284761 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:15:00.284733 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb143d6a_6dda_40db_8e3f_f64558e4fec5.slice/crio-768ed259c951d7e3b8a725906bc17b8154147f4decf237bd5c9382dab4444d35 WatchSource:0}: Error finding container 768ed259c951d7e3b8a725906bc17b8154147f4decf237bd5c9382dab4444d35: Status 404 returned error can't find the container with id 768ed259c951d7e3b8a725906bc17b8154147f4decf237bd5c9382dab4444d35 Jul 15 15:15:00.559014 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:00.558984 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0c81d7-29735475-nbppm" event={"ID":"db143d6a-6dda-40db-8e3f-f64558e4fec5","Type":"ContainerStarted","Data":"768ed259c951d7e3b8a725906bc17b8154147f4decf237bd5c9382dab4444d35"} Jul 15 15:15:01.566197 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:01.566160 2574 generic.go:356] "Generic (PLEG): container finished" podID="db143d6a-6dda-40db-8e3f-f64558e4fec5" containerID="ff2a70ba696510040872b6869ac6c0935b8d9bc0ce2ca4101aba4cf7d78a7dcf" exitCode=0 Jul 15 15:15:01.566197 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:01.566196 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0c81d7-29735475-nbppm" event={"ID":"db143d6a-6dda-40db-8e3f-f64558e4fec5","Type":"ContainerDied","Data":"ff2a70ba696510040872b6869ac6c0935b8d9bc0ce2ca4101aba4cf7d78a7dcf"} Jul 15 15:15:02.702693 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:02.702670 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0c81d7-29735475-nbppm" Jul 15 15:15:03.577186 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:03.577149 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0c81d7-29735475-nbppm" event={"ID":"db143d6a-6dda-40db-8e3f-f64558e4fec5","Type":"ContainerDied","Data":"768ed259c951d7e3b8a725906bc17b8154147f4decf237bd5c9382dab4444d35"} Jul 15 15:15:03.577186 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:03.577179 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0c81d7-29735475-nbppm" Jul 15 15:15:03.577445 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:03.577199 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="768ed259c951d7e3b8a725906bc17b8154147f4decf237bd5c9382dab4444d35" Jul 15 15:15:22.620657 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.620625 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk"] Jul 15 15:15:22.621083 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.621016 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db143d6a-6dda-40db-8e3f-f64558e4fec5" containerName="cleanup" Jul 15 15:15:22.621083 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.621033 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="db143d6a-6dda-40db-8e3f-f64558e4fec5" containerName="cleanup" Jul 15 15:15:22.621159 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.621091 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="db143d6a-6dda-40db-8e3f-f64558e4fec5" containerName="cleanup" Jul 15 15:15:22.623090 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.623072 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.625573 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.625553 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0c81d7\"/\"default-dockercfg-67sdb\"" Jul 15 15:15:22.626752 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.626727 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0c81d7\"/\"auth-test-model-0c81d7-kserve-self-signed-certs\"" Jul 15 15:15:22.626867 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.626754 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0c81d7\"/\"kube-root-ca.crt\"" Jul 15 15:15:22.626867 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.626772 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0c81d7\"/\"openshift-service-ca.crt\"" Jul 15 15:15:22.632532 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.632510 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk"] Jul 15 15:15:22.685695 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.685670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b434aec7-8405-4fce-b5fd-981e44f7b123-tls-certs\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.685843 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.685701 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-tmp-dir\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.685843 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.685754 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-dshm\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.685843 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.685770 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-home\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.685843 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.685789 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-model-cache\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.685843 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.685805 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4rcd\" (UniqueName: \"kubernetes.io/projected/b434aec7-8405-4fce-b5fd-981e44f7b123-kube-api-access-k4rcd\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.786878 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.786848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-dshm\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.786878 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.786902 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-home\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.787110 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.786922 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-model-cache\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.787110 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.786938 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4rcd\" (UniqueName: \"kubernetes.io/projected/b434aec7-8405-4fce-b5fd-981e44f7b123-kube-api-access-k4rcd\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.787110 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.786980 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b434aec7-8405-4fce-b5fd-981e44f7b123-tls-certs\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.787110 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.786997 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-tmp-dir\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.787343 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.787322 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-home\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.787440 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.787375 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-model-cache\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.787440 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.787411 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-tmp-dir\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.788989 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.788968 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-dshm\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.789418 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.789396 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b434aec7-8405-4fce-b5fd-981e44f7b123-tls-certs\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.796484 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.796462 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4rcd\" (UniqueName: \"kubernetes.io/projected/b434aec7-8405-4fce-b5fd-981e44f7b123-kube-api-access-k4rcd\") pod \"auth-test-model-0c81d7-kserve-cd7f89459-srtlk\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:22.932955 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:22.932836 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:23.063464 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:23.063439 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk"] Jul 15 15:15:23.065112 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:15:23.065086 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb434aec7_8405_4fce_b5fd_981e44f7b123.slice/crio-8a445159681b95e468f0303646dc968674127df5c0cd2ddb75fdfecbeee07e2d WatchSource:0}: Error finding container 8a445159681b95e468f0303646dc968674127df5c0cd2ddb75fdfecbeee07e2d: Status 404 returned error can't find the container with id 8a445159681b95e468f0303646dc968674127df5c0cd2ddb75fdfecbeee07e2d Jul 15 15:15:23.675602 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:23.675563 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" event={"ID":"b434aec7-8405-4fce-b5fd-981e44f7b123","Type":"ContainerStarted","Data":"dd6f0c1a5ee2baa889b31e2c2f083980706171115e545adecd48f35d3d797179"} Jul 15 15:15:23.675602 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:23.675604 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" event={"ID":"b434aec7-8405-4fce-b5fd-981e44f7b123","Type":"ContainerStarted","Data":"8a445159681b95e468f0303646dc968674127df5c0cd2ddb75fdfecbeee07e2d"} Jul 15 15:15:23.696545 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:23.696496 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" podStartSLOduration=1.696480856 podStartE2EDuration="1.696480856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:15:23.695177781 +0000 UTC m=+2496.401253887" watchObservedRunningTime="2026-07-15 15:15:23.696480856 +0000 UTC m=+2496.402556988" Jul 15 15:15:32.933192 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:32.933157 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:32.933718 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:32.933205 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:32.946187 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:32.946167 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:15:33.732489 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:15:33.732459 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:17:58.334235 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:17:58.334201 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk"] Jul 15 15:18:13.353918 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.353845 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" podUID="b434aec7-8405-4fce-b5fd-981e44f7b123" containerName="main" containerID="cri-o://dd6f0c1a5ee2baa889b31e2c2f083980706171115e545adecd48f35d3d797179" gracePeriod=45 Jul 15 15:18:13.511446 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.511416 2574 generic.go:356] "Generic (PLEG): container finished" podID="b434aec7-8405-4fce-b5fd-981e44f7b123" containerID="dd6f0c1a5ee2baa889b31e2c2f083980706171115e545adecd48f35d3d797179" exitCode=0 Jul 15 15:18:13.511590 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.511453 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" event={"ID":"b434aec7-8405-4fce-b5fd-981e44f7b123","Type":"ContainerDied","Data":"dd6f0c1a5ee2baa889b31e2c2f083980706171115e545adecd48f35d3d797179"} Jul 15 15:18:13.595359 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.595336 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:18:13.748185 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.748092 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4rcd\" (UniqueName: \"kubernetes.io/projected/b434aec7-8405-4fce-b5fd-981e44f7b123-kube-api-access-k4rcd\") pod \"b434aec7-8405-4fce-b5fd-981e44f7b123\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " Jul 15 15:18:13.748185 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.748152 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-home\") pod \"b434aec7-8405-4fce-b5fd-981e44f7b123\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " Jul 15 15:18:13.748385 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.748350 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b434aec7-8405-4fce-b5fd-981e44f7b123-tls-certs\") pod \"b434aec7-8405-4fce-b5fd-981e44f7b123\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " Jul 15 15:18:13.748422 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.748385 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-model-cache\") pod \"b434aec7-8405-4fce-b5fd-981e44f7b123\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " Jul 15 15:18:13.748422 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.748398 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-home" (OuterVolumeSpecName: "home") pod "b434aec7-8405-4fce-b5fd-981e44f7b123" (UID: "b434aec7-8405-4fce-b5fd-981e44f7b123"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:18:13.748502 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.748427 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-tmp-dir\") pod \"b434aec7-8405-4fce-b5fd-981e44f7b123\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " Jul 15 15:18:13.748502 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.748471 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-dshm\") pod \"b434aec7-8405-4fce-b5fd-981e44f7b123\" (UID: \"b434aec7-8405-4fce-b5fd-981e44f7b123\") " Jul 15 15:18:13.748617 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.748582 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-model-cache" (OuterVolumeSpecName: "model-cache") pod "b434aec7-8405-4fce-b5fd-981e44f7b123" (UID: "b434aec7-8405-4fce-b5fd-981e44f7b123"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:18:13.748680 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.748652 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b434aec7-8405-4fce-b5fd-981e44f7b123" (UID: "b434aec7-8405-4fce-b5fd-981e44f7b123"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:18:13.748819 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.748801 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-home\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:18:13.748874 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.748819 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-model-cache\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:18:13.748874 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.748829 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-tmp-dir\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:18:13.750443 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.750423 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b434aec7-8405-4fce-b5fd-981e44f7b123-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b434aec7-8405-4fce-b5fd-981e44f7b123" (UID: "b434aec7-8405-4fce-b5fd-981e44f7b123"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 15:18:13.750524 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.750507 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b434aec7-8405-4fce-b5fd-981e44f7b123-kube-api-access-k4rcd" (OuterVolumeSpecName: "kube-api-access-k4rcd") pod "b434aec7-8405-4fce-b5fd-981e44f7b123" (UID: "b434aec7-8405-4fce-b5fd-981e44f7b123"). InnerVolumeSpecName "kube-api-access-k4rcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:18:13.750562 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.750509 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-dshm" (OuterVolumeSpecName: "dshm") pod "b434aec7-8405-4fce-b5fd-981e44f7b123" (UID: "b434aec7-8405-4fce-b5fd-981e44f7b123"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:18:13.849945 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.849914 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4rcd\" (UniqueName: \"kubernetes.io/projected/b434aec7-8405-4fce-b5fd-981e44f7b123-kube-api-access-k4rcd\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:18:13.849945 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.849940 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b434aec7-8405-4fce-b5fd-981e44f7b123-tls-certs\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:18:13.849945 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:13.849951 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b434aec7-8405-4fce-b5fd-981e44f7b123-dshm\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:18:14.517262 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:14.517232 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" Jul 15 15:18:14.517667 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:14.517230 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk" event={"ID":"b434aec7-8405-4fce-b5fd-981e44f7b123","Type":"ContainerDied","Data":"8a445159681b95e468f0303646dc968674127df5c0cd2ddb75fdfecbeee07e2d"} Jul 15 15:18:14.517667 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:14.517363 2574 scope.go:117] "RemoveContainer" containerID="dd6f0c1a5ee2baa889b31e2c2f083980706171115e545adecd48f35d3d797179" Jul 15 15:18:14.540837 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:14.540813 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk"] Jul 15 15:18:14.545096 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:14.545076 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0c81d7/auth-test-model-0c81d7-kserve-cd7f89459-srtlk"] Jul 15 15:18:15.083701 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:15.083666 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b434aec7-8405-4fce-b5fd-981e44f7b123" path="/var/lib/kubelet/pods/b434aec7-8405-4fce-b5fd-981e44f7b123/volumes" Jul 15 15:18:38.313156 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.310599 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94"] Jul 15 15:18:38.313156 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.311430 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b434aec7-8405-4fce-b5fd-981e44f7b123" containerName="main" Jul 15 15:18:38.313156 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.311449 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b434aec7-8405-4fce-b5fd-981e44f7b123" containerName="main" Jul 15 15:18:38.313156 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.311643 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b434aec7-8405-4fce-b5fd-981e44f7b123" containerName="main" Jul 15 15:18:38.315578 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.315560 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.319551 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.319519 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b2aed6\"/\"default-dockercfg-kvwnr\"" Jul 15 15:18:38.319551 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.319531 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b2aed6\"/\"sub-test-model-b2aed6-kserve-self-signed-certs\"" Jul 15 15:18:38.319551 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.319529 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b2aed6\"/\"openshift-service-ca.crt\"" Jul 15 15:18:38.319803 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.319529 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b2aed6\"/\"kube-root-ca.crt\"" Jul 15 15:18:38.320555 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.320533 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94"] Jul 15 15:18:38.354935 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.354908 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdpsj\" (UniqueName: \"kubernetes.io/projected/956d7e12-6f9d-4807-852c-b123a7f2dfdb-kube-api-access-mdpsj\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.355043 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.354965 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-home\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.355043 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.355012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956d7e12-6f9d-4807-852c-b123a7f2dfdb-tls-certs\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.355157 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.355133 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-tmp-dir\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.355213 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.355198 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-dshm\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.355272 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.355258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-model-cache\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.456286 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.456251 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdpsj\" (UniqueName: \"kubernetes.io/projected/956d7e12-6f9d-4807-852c-b123a7f2dfdb-kube-api-access-mdpsj\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.456430 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.456316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-home\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.456499 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.456477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956d7e12-6f9d-4807-852c-b123a7f2dfdb-tls-certs\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.456563 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.456534 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-tmp-dir\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.456620 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.456586 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-dshm\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.456682 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.456639 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-model-cache\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.456741 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.456700 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-home\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.456805 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.456787 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-tmp-dir\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.456996 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.456980 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-model-cache\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.458738 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.458712 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-dshm\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.458909 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.458875 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956d7e12-6f9d-4807-852c-b123a7f2dfdb-tls-certs\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.464876 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.464859 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdpsj\" (UniqueName: \"kubernetes.io/projected/956d7e12-6f9d-4807-852c-b123a7f2dfdb-kube-api-access-mdpsj\") pod \"sub-test-model-b2aed6-kserve-7f9858c6f9-vch94\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.625482 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.625397 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:38.752798 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:38.752770 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94"] Jul 15 15:18:38.754879 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:18:38.754853 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod956d7e12_6f9d_4807_852c_b123a7f2dfdb.slice/crio-3cf413fda74d4bacb10125dfc5998effcd05b35c0952ed52448dc09fb16ae0e1 WatchSource:0}: Error finding container 3cf413fda74d4bacb10125dfc5998effcd05b35c0952ed52448dc09fb16ae0e1: Status 404 returned error can't find the container with id 3cf413fda74d4bacb10125dfc5998effcd05b35c0952ed52448dc09fb16ae0e1 Jul 15 15:18:39.643131 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:39.643050 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" event={"ID":"956d7e12-6f9d-4807-852c-b123a7f2dfdb","Type":"ContainerStarted","Data":"4e2c1f191d32624f853b54d16bb4c8a3913553a032972d46ed7b975deab90b67"} Jul 15 15:18:39.643131 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:39.643099 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" event={"ID":"956d7e12-6f9d-4807-852c-b123a7f2dfdb","Type":"ContainerStarted","Data":"3cf413fda74d4bacb10125dfc5998effcd05b35c0952ed52448dc09fb16ae0e1"} Jul 15 15:18:39.661074 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:39.661030 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" podStartSLOduration=1.6610144500000001 podStartE2EDuration="1.66101445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:18:39.660138127 +0000 UTC m=+2692.366214232" watchObservedRunningTime="2026-07-15 15:18:39.66101445 +0000 UTC m=+2692.367090554" Jul 15 15:18:48.626086 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:48.626047 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:48.626086 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:48.626098 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:48.638868 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:48.638843 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:48.698080 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:48.698045 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:18:49.094396 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:49.094360 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 15:18:49.100719 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:18:49.100697 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bdhfs_ee8a402a-6831-434d-b19d-d0a8617c5c91/ovn-acl-logging/0.log" Jul 15 15:19:29.879748 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:29.879717 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94"] Jul 15 15:19:44.901273 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:44.901096 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" podUID="956d7e12-6f9d-4807-852c-b123a7f2dfdb" containerName="main" containerID="cri-o://4e2c1f191d32624f853b54d16bb4c8a3913553a032972d46ed7b975deab90b67" gracePeriod=45 Jul 15 15:19:45.151486 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.151429 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:19:45.223422 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.223397 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdpsj\" (UniqueName: \"kubernetes.io/projected/956d7e12-6f9d-4807-852c-b123a7f2dfdb-kube-api-access-mdpsj\") pod \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " Jul 15 15:19:45.223560 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.223431 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-dshm\") pod \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " Jul 15 15:19:45.223560 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.223454 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956d7e12-6f9d-4807-852c-b123a7f2dfdb-tls-certs\") pod \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " Jul 15 15:19:45.223560 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.223528 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-tmp-dir\") pod \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " Jul 15 15:19:45.223560 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.223554 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-home\") pod \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " Jul 15 15:19:45.223746 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.223580 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-model-cache\") pod \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\" (UID: \"956d7e12-6f9d-4807-852c-b123a7f2dfdb\") " Jul 15 15:19:45.223805 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.223753 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "956d7e12-6f9d-4807-852c-b123a7f2dfdb" (UID: "956d7e12-6f9d-4807-852c-b123a7f2dfdb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:19:45.223805 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.223787 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-home" (OuterVolumeSpecName: "home") pod "956d7e12-6f9d-4807-852c-b123a7f2dfdb" (UID: "956d7e12-6f9d-4807-852c-b123a7f2dfdb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:19:45.223942 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.223858 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-model-cache" (OuterVolumeSpecName: "model-cache") pod "956d7e12-6f9d-4807-852c-b123a7f2dfdb" (UID: "956d7e12-6f9d-4807-852c-b123a7f2dfdb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:19:45.224001 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.223983 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-tmp-dir\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:19:45.224039 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.224007 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-home\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:19:45.224039 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.224021 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-model-cache\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:19:45.225636 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.225614 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-dshm" (OuterVolumeSpecName: "dshm") pod "956d7e12-6f9d-4807-852c-b123a7f2dfdb" (UID: "956d7e12-6f9d-4807-852c-b123a7f2dfdb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:19:45.225719 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.225703 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956d7e12-6f9d-4807-852c-b123a7f2dfdb-kube-api-access-mdpsj" (OuterVolumeSpecName: "kube-api-access-mdpsj") pod "956d7e12-6f9d-4807-852c-b123a7f2dfdb" (UID: "956d7e12-6f9d-4807-852c-b123a7f2dfdb"). InnerVolumeSpecName "kube-api-access-mdpsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:19:45.225760 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.225745 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/956d7e12-6f9d-4807-852c-b123a7f2dfdb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "956d7e12-6f9d-4807-852c-b123a7f2dfdb" (UID: "956d7e12-6f9d-4807-852c-b123a7f2dfdb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 15:19:45.325397 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.325372 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdpsj\" (UniqueName: \"kubernetes.io/projected/956d7e12-6f9d-4807-852c-b123a7f2dfdb-kube-api-access-mdpsj\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:19:45.325397 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.325394 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/956d7e12-6f9d-4807-852c-b123a7f2dfdb-dshm\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:19:45.325545 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.325405 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956d7e12-6f9d-4807-852c-b123a7f2dfdb-tls-certs\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:19:45.973703 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.973667 2574 generic.go:356] "Generic (PLEG): container finished" podID="956d7e12-6f9d-4807-852c-b123a7f2dfdb" containerID="4e2c1f191d32624f853b54d16bb4c8a3913553a032972d46ed7b975deab90b67" exitCode=0 Jul 15 15:19:45.973703 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.973703 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" Jul 15 15:19:45.974228 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.973761 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" event={"ID":"956d7e12-6f9d-4807-852c-b123a7f2dfdb","Type":"ContainerDied","Data":"4e2c1f191d32624f853b54d16bb4c8a3913553a032972d46ed7b975deab90b67"} Jul 15 15:19:45.974228 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.973811 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94" event={"ID":"956d7e12-6f9d-4807-852c-b123a7f2dfdb","Type":"ContainerDied","Data":"3cf413fda74d4bacb10125dfc5998effcd05b35c0952ed52448dc09fb16ae0e1"} Jul 15 15:19:45.974228 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.973834 2574 scope.go:117] "RemoveContainer" containerID="4e2c1f191d32624f853b54d16bb4c8a3913553a032972d46ed7b975deab90b67" Jul 15 15:19:45.983723 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.983708 2574 scope.go:117] "RemoveContainer" containerID="4e2c1f191d32624f853b54d16bb4c8a3913553a032972d46ed7b975deab90b67" Jul 15 15:19:45.984003 ip-10-0-139-83 kubenswrapper[2574]: E0715 15:19:45.983984 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2c1f191d32624f853b54d16bb4c8a3913553a032972d46ed7b975deab90b67\": container with ID starting with 4e2c1f191d32624f853b54d16bb4c8a3913553a032972d46ed7b975deab90b67 not found: ID does not exist" containerID="4e2c1f191d32624f853b54d16bb4c8a3913553a032972d46ed7b975deab90b67" Jul 15 15:19:45.984060 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:45.984011 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2c1f191d32624f853b54d16bb4c8a3913553a032972d46ed7b975deab90b67"} err="failed to get container status \"4e2c1f191d32624f853b54d16bb4c8a3913553a032972d46ed7b975deab90b67\": rpc error: code = NotFound desc = could not find container \"4e2c1f191d32624f853b54d16bb4c8a3913553a032972d46ed7b975deab90b67\": container with ID starting with 4e2c1f191d32624f853b54d16bb4c8a3913553a032972d46ed7b975deab90b67 not found: ID does not exist" Jul 15 15:19:46.000186 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:46.000147 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94"] Jul 15 15:19:46.003919 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:46.003878 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b2aed6/sub-test-model-b2aed6-kserve-7f9858c6f9-vch94"] Jul 15 15:19:47.083525 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:19:47.083482 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="956d7e12-6f9d-4807-852c-b123a7f2dfdb" path="/var/lib/kubelet/pods/956d7e12-6f9d-4807-852c-b123a7f2dfdb/volumes" Jul 15 15:20:05.100191 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.100145 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n"] Jul 15 15:20:05.100768 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.100508 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="956d7e12-6f9d-4807-852c-b123a7f2dfdb" containerName="main" Jul 15 15:20:05.100768 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.100518 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="956d7e12-6f9d-4807-852c-b123a7f2dfdb" containerName="main" Jul 15 15:20:05.100768 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.100576 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="956d7e12-6f9d-4807-852c-b123a7f2dfdb" containerName="main" Jul 15 15:20:05.103711 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.103690 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.108240 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.108219 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b2aed6\"/\"openshift-service-ca.crt\"" Jul 15 15:20:05.108240 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.108232 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b2aed6\"/\"default-dockercfg-kvwnr\"" Jul 15 15:20:05.108422 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.108256 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b2aed6\"/\"kube-root-ca.crt\"" Jul 15 15:20:05.108422 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.108234 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b2aed6\"/\"rate-test-model-b2aed6-kserve-self-signed-certs\"" Jul 15 15:20:05.112077 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.112057 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n"] Jul 15 15:20:05.294606 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.294570 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-dshm\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.294776 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.294611 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-tmp-dir\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.294776 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.294634 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd51173-94e4-4aaa-b910-73ee165a201f-tls-certs\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.294776 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.294712 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtw4f\" (UniqueName: \"kubernetes.io/projected/fbd51173-94e4-4aaa-b910-73ee165a201f-kube-api-access-qtw4f\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.294776 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.294752 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-home\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.294776 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.294771 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-model-cache\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.395414 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.395341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-tmp-dir\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.395414 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.395369 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd51173-94e4-4aaa-b910-73ee165a201f-tls-certs\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.395414 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.395390 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtw4f\" (UniqueName: \"kubernetes.io/projected/fbd51173-94e4-4aaa-b910-73ee165a201f-kube-api-access-qtw4f\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.395606 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.395509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-home\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.395606 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.395550 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-model-cache\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.395700 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.395684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-dshm\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.395769 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.395729 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-tmp-dir\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.395848 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.395806 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-home\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.395987 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.395967 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-model-cache\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.397686 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.397668 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-dshm\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.397790 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.397773 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd51173-94e4-4aaa-b910-73ee165a201f-tls-certs\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.404566 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.404546 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtw4f\" (UniqueName: \"kubernetes.io/projected/fbd51173-94e4-4aaa-b910-73ee165a201f-kube-api-access-qtw4f\") pod \"rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.414457 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.414437 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:05.547618 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:05.547594 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n"] Jul 15 15:20:05.549655 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:20:05.549628 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbd51173_94e4_4aaa_b910_73ee165a201f.slice/crio-7385c665d371d17b795c895367f2645a0cb9b8907b578ed3855ea0cc103e5f00 WatchSource:0}: Error finding container 7385c665d371d17b795c895367f2645a0cb9b8907b578ed3855ea0cc103e5f00: Status 404 returned error can't find the container with id 7385c665d371d17b795c895367f2645a0cb9b8907b578ed3855ea0cc103e5f00 Jul 15 15:20:06.076080 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:06.076011 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" event={"ID":"fbd51173-94e4-4aaa-b910-73ee165a201f","Type":"ContainerStarted","Data":"1ecf354e07286fad146182b32e078da2f3b246878c84772f911d6314d282ff2d"} Jul 15 15:20:06.076260 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:06.076090 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" event={"ID":"fbd51173-94e4-4aaa-b910-73ee165a201f","Type":"ContainerStarted","Data":"7385c665d371d17b795c895367f2645a0cb9b8907b578ed3855ea0cc103e5f00"} Jul 15 15:20:06.096526 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:06.096478 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" podStartSLOduration=1.096464197 podStartE2EDuration="1.096464197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 15:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:20:06.094455011 +0000 UTC m=+2778.800531115" watchObservedRunningTime="2026-07-15 15:20:06.096464197 +0000 UTC m=+2778.802540302" Jul 15 15:20:15.415449 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:15.415364 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:15.415449 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:15.415420 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:15.427766 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:15.427743 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:16.136606 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:16.136576 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:20:52.678329 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:52.678294 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n"] Jul 15 15:20:56.313623 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:56.313592 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7179-140050791951664"] Jul 15 15:20:56.316869 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:56.316843 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7179-140050791951664" Jul 15 15:20:56.319288 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:56.319269 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tkmgr\"" Jul 15 15:20:56.324116 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:56.324093 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7179-140050791951664"] Jul 15 15:20:56.426562 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:56.426525 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjjt9\" (UniqueName: \"kubernetes.io/projected/fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91-kube-api-access-qjjt9\") pod \"test-curl-7179-140050791951664\" (UID: \"fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91\") " pod="opendatahub/test-curl-7179-140050791951664" Jul 15 15:20:56.527640 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:56.527610 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjjt9\" (UniqueName: \"kubernetes.io/projected/fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91-kube-api-access-qjjt9\") pod \"test-curl-7179-140050791951664\" (UID: \"fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91\") " pod="opendatahub/test-curl-7179-140050791951664" Jul 15 15:20:56.537039 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:56.537004 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjjt9\" (UniqueName: \"kubernetes.io/projected/fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91-kube-api-access-qjjt9\") pod \"test-curl-7179-140050791951664\" (UID: \"fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91\") " pod="opendatahub/test-curl-7179-140050791951664" Jul 15 15:20:56.628619 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:56.628551 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7179-140050791951664" Jul 15 15:20:56.757696 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:56.757665 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7179-140050791951664"] Jul 15 15:20:56.759656 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:20:56.759626 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbdbf896_cb4a_4ad0_8efc_fc20c3c94d91.slice/crio-6c9f3929ab54bb500303ae29a3916554b8960df4a7a805ddf06034cc138450d7 WatchSource:0}: Error finding container 6c9f3929ab54bb500303ae29a3916554b8960df4a7a805ddf06034cc138450d7: Status 404 returned error can't find the container with id 6c9f3929ab54bb500303ae29a3916554b8960df4a7a805ddf06034cc138450d7 Jul 15 15:20:56.760844 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:56.760827 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 15:20:57.330180 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:57.330145 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7179-140050791951664" event={"ID":"fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91","Type":"ContainerStarted","Data":"6c9f3929ab54bb500303ae29a3916554b8960df4a7a805ddf06034cc138450d7"} Jul 15 15:20:58.336158 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:58.336123 2574 generic.go:356] "Generic (PLEG): container finished" podID="fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91" containerID="18c4aae2937c102bcace51e709a0aad74835f84423f667e31bef22cc6bce6ff3" exitCode=0 Jul 15 15:20:58.336555 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:58.336220 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7179-140050791951664" event={"ID":"fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91","Type":"ContainerDied","Data":"18c4aae2937c102bcace51e709a0aad74835f84423f667e31bef22cc6bce6ff3"} Jul 15 15:20:59.474178 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:59.474152 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7179-140050791951664" Jul 15 15:20:59.532789 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:59.532762 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7179-140050791951664_fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91/test-curl-7179-140050791951664/0.log" Jul 15 15:20:59.563459 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:59.563426 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7179-140050791951664"] Jul 15 15:20:59.568233 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:59.568214 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7179-140050791951664"] Jul 15 15:20:59.658584 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:59.658513 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjjt9\" (UniqueName: \"kubernetes.io/projected/fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91-kube-api-access-qjjt9\") pod \"fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91\" (UID: \"fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91\") " Jul 15 15:20:59.660996 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:59.660972 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91-kube-api-access-qjjt9" (OuterVolumeSpecName: "kube-api-access-qjjt9") pod "fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91" (UID: "fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91"). InnerVolumeSpecName "kube-api-access-qjjt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:20:59.759831 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:20:59.759806 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjjt9\" (UniqueName: \"kubernetes.io/projected/fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91-kube-api-access-qjjt9\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:21:00.350733 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:00.350704 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7179-140050791951664" Jul 15 15:21:00.350733 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:00.350733 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c9f3929ab54bb500303ae29a3916554b8960df4a7a805ddf06034cc138450d7" Jul 15 15:21:00.362609 ip-10-0-139-83 kubenswrapper[2574]: E0715 15:21:00.362578 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7179-140050791951664\" is forbidden: User \"system:node:ip-10-0-139-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-83.ec2.internal' and this object" podUID="fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91" pod="opendatahub/test-curl-7179-140050791951664" Jul 15 15:21:01.084233 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:01.084191 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91" path="/var/lib/kubelet/pods/fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91/volumes" Jul 15 15:21:02.365558 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:02.365523 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7179-140050791854128"] Jul 15 15:21:02.365936 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:02.365874 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91" containerName="test-curl-7179-140050791951664" Jul 15 15:21:02.365936 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:02.365899 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91" containerName="test-curl-7179-140050791951664" Jul 15 15:21:02.366015 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:02.365964 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbdbf896-cb4a-4ad0-8efc-fc20c3c94d91" containerName="test-curl-7179-140050791951664" Jul 15 15:21:02.368874 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:02.368858 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7179-140050791854128" Jul 15 15:21:02.371214 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:02.371193 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tkmgr\"" Jul 15 15:21:02.375441 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:02.375415 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7179-140050791854128"] Jul 15 15:21:02.385001 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:02.384980 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwhw2\" (UniqueName: \"kubernetes.io/projected/4ef58a8d-36c6-4aee-8572-04d907d1ba84-kube-api-access-hwhw2\") pod \"test-curl-7179-140050791854128\" (UID: \"4ef58a8d-36c6-4aee-8572-04d907d1ba84\") " pod="opendatahub/test-curl-7179-140050791854128" Jul 15 15:21:02.485608 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:02.485585 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwhw2\" (UniqueName: \"kubernetes.io/projected/4ef58a8d-36c6-4aee-8572-04d907d1ba84-kube-api-access-hwhw2\") pod \"test-curl-7179-140050791854128\" (UID: \"4ef58a8d-36c6-4aee-8572-04d907d1ba84\") " pod="opendatahub/test-curl-7179-140050791854128" Jul 15 15:21:02.494865 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:02.494839 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwhw2\" (UniqueName: \"kubernetes.io/projected/4ef58a8d-36c6-4aee-8572-04d907d1ba84-kube-api-access-hwhw2\") pod \"test-curl-7179-140050791854128\" (UID: \"4ef58a8d-36c6-4aee-8572-04d907d1ba84\") " pod="opendatahub/test-curl-7179-140050791854128" Jul 15 15:21:02.678549 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:02.678461 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7179-140050791854128" Jul 15 15:21:02.805418 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:02.805392 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7179-140050791854128"] Jul 15 15:21:02.807440 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:21:02.807415 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ef58a8d_36c6_4aee_8572_04d907d1ba84.slice/crio-89f9158dcde507366bb144dc5c89c0d125f8d7340572fd2412460bad6feeaa50 WatchSource:0}: Error finding container 89f9158dcde507366bb144dc5c89c0d125f8d7340572fd2412460bad6feeaa50: Status 404 returned error can't find the container with id 89f9158dcde507366bb144dc5c89c0d125f8d7340572fd2412460bad6feeaa50 Jul 15 15:21:03.367504 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:03.367475 2574 generic.go:356] "Generic (PLEG): container finished" podID="4ef58a8d-36c6-4aee-8572-04d907d1ba84" containerID="df90e4716c8325f25474dc3c3090abb80c5248efd4931700169ffabcb7c0484c" exitCode=0 Jul 15 15:21:03.367869 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:03.367557 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7179-140050791854128" event={"ID":"4ef58a8d-36c6-4aee-8572-04d907d1ba84","Type":"ContainerDied","Data":"df90e4716c8325f25474dc3c3090abb80c5248efd4931700169ffabcb7c0484c"} Jul 15 15:21:03.367869 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:03.367600 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7179-140050791854128" event={"ID":"4ef58a8d-36c6-4aee-8572-04d907d1ba84","Type":"ContainerStarted","Data":"89f9158dcde507366bb144dc5c89c0d125f8d7340572fd2412460bad6feeaa50"} Jul 15 15:21:04.501013 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:04.500988 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7179-140050791854128" Jul 15 15:21:04.523257 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:04.523232 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7179-140050791854128_4ef58a8d-36c6-4aee-8572-04d907d1ba84/test-curl-7179-140050791854128/0.log" Jul 15 15:21:04.548553 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:04.548531 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7179-140050791854128"] Jul 15 15:21:04.552624 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:04.552599 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7179-140050791854128"] Jul 15 15:21:04.605632 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:04.605607 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwhw2\" (UniqueName: \"kubernetes.io/projected/4ef58a8d-36c6-4aee-8572-04d907d1ba84-kube-api-access-hwhw2\") pod \"4ef58a8d-36c6-4aee-8572-04d907d1ba84\" (UID: \"4ef58a8d-36c6-4aee-8572-04d907d1ba84\") " Jul 15 15:21:04.607528 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:04.607504 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef58a8d-36c6-4aee-8572-04d907d1ba84-kube-api-access-hwhw2" (OuterVolumeSpecName: "kube-api-access-hwhw2") pod "4ef58a8d-36c6-4aee-8572-04d907d1ba84" (UID: "4ef58a8d-36c6-4aee-8572-04d907d1ba84"). InnerVolumeSpecName "kube-api-access-hwhw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:21:04.707114 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:04.707054 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwhw2\" (UniqueName: \"kubernetes.io/projected/4ef58a8d-36c6-4aee-8572-04d907d1ba84-kube-api-access-hwhw2\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:21:05.083102 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:05.083068 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ef58a8d-36c6-4aee-8572-04d907d1ba84" path="/var/lib/kubelet/pods/4ef58a8d-36c6-4aee-8572-04d907d1ba84/volumes" Jul 15 15:21:05.377751 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:05.377666 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7179-140050791854128" Jul 15 15:21:05.377751 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:05.377715 2574 scope.go:117] "RemoveContainer" containerID="df90e4716c8325f25474dc3c3090abb80c5248efd4931700169ffabcb7c0484c" Jul 15 15:21:07.698323 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:07.698275 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" podUID="fbd51173-94e4-4aaa-b910-73ee165a201f" containerName="main" containerID="cri-o://1ecf354e07286fad146182b32e078da2f3b246878c84772f911d6314d282ff2d" gracePeriod=45 Jul 15 15:21:07.953199 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:07.953141 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:21:08.031123 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.031099 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-model-cache\") pod \"fbd51173-94e4-4aaa-b910-73ee165a201f\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " Jul 15 15:21:08.031278 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.031146 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtw4f\" (UniqueName: \"kubernetes.io/projected/fbd51173-94e4-4aaa-b910-73ee165a201f-kube-api-access-qtw4f\") pod \"fbd51173-94e4-4aaa-b910-73ee165a201f\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " Jul 15 15:21:08.031278 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.031212 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-home\") pod \"fbd51173-94e4-4aaa-b910-73ee165a201f\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " Jul 15 15:21:08.031278 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.031242 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd51173-94e4-4aaa-b910-73ee165a201f-tls-certs\") pod \"fbd51173-94e4-4aaa-b910-73ee165a201f\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " Jul 15 15:21:08.031437 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.031294 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-dshm\") pod \"fbd51173-94e4-4aaa-b910-73ee165a201f\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " Jul 15 15:21:08.031437 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.031318 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-tmp-dir\") pod \"fbd51173-94e4-4aaa-b910-73ee165a201f\" (UID: \"fbd51173-94e4-4aaa-b910-73ee165a201f\") " Jul 15 15:21:08.031437 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.031330 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-model-cache" (OuterVolumeSpecName: "model-cache") pod "fbd51173-94e4-4aaa-b910-73ee165a201f" (UID: "fbd51173-94e4-4aaa-b910-73ee165a201f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:21:08.031578 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.031454 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-home" (OuterVolumeSpecName: "home") pod "fbd51173-94e4-4aaa-b910-73ee165a201f" (UID: "fbd51173-94e4-4aaa-b910-73ee165a201f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:21:08.031636 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.031593 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-model-cache\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:21:08.031636 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.031595 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fbd51173-94e4-4aaa-b910-73ee165a201f" (UID: "fbd51173-94e4-4aaa-b910-73ee165a201f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:21:08.031636 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.031610 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-home\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:21:08.033357 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.033326 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd51173-94e4-4aaa-b910-73ee165a201f-kube-api-access-qtw4f" (OuterVolumeSpecName: "kube-api-access-qtw4f") pod "fbd51173-94e4-4aaa-b910-73ee165a201f" (UID: "fbd51173-94e4-4aaa-b910-73ee165a201f"). InnerVolumeSpecName "kube-api-access-qtw4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 15:21:08.033357 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.033326 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-dshm" (OuterVolumeSpecName: "dshm") pod "fbd51173-94e4-4aaa-b910-73ee165a201f" (UID: "fbd51173-94e4-4aaa-b910-73ee165a201f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 15:21:08.033501 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.033357 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbd51173-94e4-4aaa-b910-73ee165a201f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fbd51173-94e4-4aaa-b910-73ee165a201f" (UID: "fbd51173-94e4-4aaa-b910-73ee165a201f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 15:21:08.132560 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.132535 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbd51173-94e4-4aaa-b910-73ee165a201f-tls-certs\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:21:08.132560 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.132557 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-dshm\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:21:08.132693 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.132566 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbd51173-94e4-4aaa-b910-73ee165a201f-tmp-dir\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:21:08.132693 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.132575 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtw4f\" (UniqueName: \"kubernetes.io/projected/fbd51173-94e4-4aaa-b910-73ee165a201f-kube-api-access-qtw4f\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:21:08.395170 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.395135 2574 generic.go:356] "Generic (PLEG): container finished" podID="fbd51173-94e4-4aaa-b910-73ee165a201f" containerID="1ecf354e07286fad146182b32e078da2f3b246878c84772f911d6314d282ff2d" exitCode=0 Jul 15 15:21:08.395290 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.395187 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" Jul 15 15:21:08.395290 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.395220 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" event={"ID":"fbd51173-94e4-4aaa-b910-73ee165a201f","Type":"ContainerDied","Data":"1ecf354e07286fad146182b32e078da2f3b246878c84772f911d6314d282ff2d"} Jul 15 15:21:08.395290 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.395264 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n" event={"ID":"fbd51173-94e4-4aaa-b910-73ee165a201f","Type":"ContainerDied","Data":"7385c665d371d17b795c895367f2645a0cb9b8907b578ed3855ea0cc103e5f00"} Jul 15 15:21:08.395290 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.395286 2574 scope.go:117] "RemoveContainer" containerID="1ecf354e07286fad146182b32e078da2f3b246878c84772f911d6314d282ff2d" Jul 15 15:21:08.404114 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.404099 2574 scope.go:117] "RemoveContainer" containerID="1ecf354e07286fad146182b32e078da2f3b246878c84772f911d6314d282ff2d" Jul 15 15:21:08.404362 ip-10-0-139-83 kubenswrapper[2574]: E0715 15:21:08.404346 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ecf354e07286fad146182b32e078da2f3b246878c84772f911d6314d282ff2d\": container with ID starting with 1ecf354e07286fad146182b32e078da2f3b246878c84772f911d6314d282ff2d not found: ID does not exist" containerID="1ecf354e07286fad146182b32e078da2f3b246878c84772f911d6314d282ff2d" Jul 15 15:21:08.404414 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.404368 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ecf354e07286fad146182b32e078da2f3b246878c84772f911d6314d282ff2d"} err="failed to get container status \"1ecf354e07286fad146182b32e078da2f3b246878c84772f911d6314d282ff2d\": rpc error: code = NotFound desc = could not find container \"1ecf354e07286fad146182b32e078da2f3b246878c84772f911d6314d282ff2d\": container with ID starting with 1ecf354e07286fad146182b32e078da2f3b246878c84772f911d6314d282ff2d not found: ID does not exist" Jul 15 15:21:08.418071 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.418049 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n"] Jul 15 15:21:08.422094 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:08.422075 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b2aed6/rate-test-model-b2aed6-kserve-76587f6bc5-dgz4n"] Jul 15 15:21:09.083468 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:09.083440 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbd51173-94e4-4aaa-b910-73ee165a201f" path="/var/lib/kubelet/pods/fbd51173-94e4-4aaa-b910-73ee165a201f/volumes" Jul 15 15:21:10.472262 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:10.472230 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb"] Jul 15 15:21:10.472625 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:10.472582 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ef58a8d-36c6-4aee-8572-04d907d1ba84" containerName="test-curl-7179-140050791854128" Jul 15 15:21:10.472625 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:10.472593 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ef58a8d-36c6-4aee-8572-04d907d1ba84" containerName="test-curl-7179-140050791854128" Jul 15 15:21:10.472625 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:10.472604 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd51173-94e4-4aaa-b910-73ee165a201f" containerName="main" Jul 15 15:21:10.472625 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:10.472609 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd51173-94e4-4aaa-b910-73ee165a201f" containerName="main" Jul 15 15:21:10.472753 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:10.472675 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ef58a8d-36c6-4aee-8572-04d907d1ba84" containerName="test-curl-7179-140050791854128" Jul 15 15:21:10.472753 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:10.472682 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbd51173-94e4-4aaa-b910-73ee165a201f" containerName="main" Jul 15 15:21:10.476977 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:10.476961 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb" Jul 15 15:21:10.479433 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:10.479407 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 15:21:10.479549 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:10.479498 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7kbgk\"" Jul 15 15:21:10.483627 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:10.483599 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb"] Jul 15 15:21:10.553037 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:10.553008 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0558d53-7e14-43a2-b621-6a61e0732bb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb\" (UID: \"e0558d53-7e14-43a2-b621-6a61e0732bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb" Jul 15 15:21:10.653558 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:10.653519 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0558d53-7e14-43a2-b621-6a61e0732bb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb\" (UID: \"e0558d53-7e14-43a2-b621-6a61e0732bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb" Jul 15 15:21:10.654145 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:10.654125 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0558d53-7e14-43a2-b621-6a61e0732bb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb\" (UID: \"e0558d53-7e14-43a2-b621-6a61e0732bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb" Jul 15 15:21:10.788941 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:10.788843 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb" Jul 15 15:21:11.124702 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:11.124674 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb"] Jul 15 15:21:11.127717 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:21:11.127690 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0558d53_7e14_43a2_b621_6a61e0732bb8.slice/crio-d6cda01c9e9ab65c3d2df13e73c513a62d86ae1605e559c413b60ad5f9143895 WatchSource:0}: Error finding container d6cda01c9e9ab65c3d2df13e73c513a62d86ae1605e559c413b60ad5f9143895: Status 404 returned error can't find the container with id d6cda01c9e9ab65c3d2df13e73c513a62d86ae1605e559c413b60ad5f9143895 Jul 15 15:21:11.412289 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:11.412224 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb" event={"ID":"e0558d53-7e14-43a2-b621-6a61e0732bb8","Type":"ContainerStarted","Data":"d6cda01c9e9ab65c3d2df13e73c513a62d86ae1605e559c413b60ad5f9143895"} Jul 15 15:21:12.418545 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:12.418509 2574 generic.go:356] "Generic (PLEG): container finished" podID="e0558d53-7e14-43a2-b621-6a61e0732bb8" containerID="dd8b572ca3c65555e47712bf5afb5a0a431395857bb364f582fc0ec4c4d0a9b9" exitCode=0 Jul 15 15:21:12.418938 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:12.418574 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb" event={"ID":"e0558d53-7e14-43a2-b621-6a61e0732bb8","Type":"ContainerDied","Data":"dd8b572ca3c65555e47712bf5afb5a0a431395857bb364f582fc0ec4c4d0a9b9"} Jul 15 15:21:13.555386 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:13.555364 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb" Jul 15 15:21:13.683775 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:13.683698 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0558d53-7e14-43a2-b621-6a61e0732bb8-maas-api-service-ca\") pod \"e0558d53-7e14-43a2-b621-6a61e0732bb8\" (UID: \"e0558d53-7e14-43a2-b621-6a61e0732bb8\") " Jul 15 15:21:13.684041 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:13.684014 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0558d53-7e14-43a2-b621-6a61e0732bb8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e0558d53-7e14-43a2-b621-6a61e0732bb8" (UID: "e0558d53-7e14-43a2-b621-6a61e0732bb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 15:21:13.784473 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:13.784451 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0558d53-7e14-43a2-b621-6a61e0732bb8-maas-api-service-ca\") on node \"ip-10-0-139-83.ec2.internal\" DevicePath \"\"" Jul 15 15:21:14.429575 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:14.429546 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb" Jul 15 15:21:14.429737 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:14.429588 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb" event={"ID":"e0558d53-7e14-43a2-b621-6a61e0732bb8","Type":"ContainerDied","Data":"d6cda01c9e9ab65c3d2df13e73c513a62d86ae1605e559c413b60ad5f9143895"} Jul 15 15:21:14.429737 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:14.429637 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6cda01c9e9ab65c3d2df13e73c513a62d86ae1605e559c413b60ad5f9143895" Jul 15 15:21:21.021763 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:21.021729 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0c81d7-29735475-nbppm"] Jul 15 15:21:21.023745 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:21.023721 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-0c81d7-29735475-nbppm"] Jul 15 15:21:21.083749 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:21.083718 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db143d6a-6dda-40db-8e3f-f64558e4fec5" path="/var/lib/kubelet/pods/db143d6a-6dda-40db-8e3f-f64558e4fec5/volumes" Jul 15 15:21:49.202255 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:21:49.202224 2574 scope.go:117] "RemoveContainer" containerID="ff2a70ba696510040872b6869ac6c0935b8d9bc0ce2ca4101aba4cf7d78a7dcf" Jul 15 15:22:03.088436 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:03.088405 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d449b7fcf-9zf46_a89662fc-7a0b-40c9-8c3b-34483f6be84c/authorino/0.log" Jul 15 15:22:09.407880 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:09.407852 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr_90bc98c4-0476-47c2-a934-24ec14e2be01/copy-manifests/0.log" Jul 15 15:22:09.415460 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:09.415440 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr_90bc98c4-0476-47c2-a934-24ec14e2be01/manager/0.log" Jul 15 15:22:10.457568 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:10.457532 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-c7c00d-hwrq9_3aac4b81-0ac8-4a9d-8725-e8715dd4ba2e/revoke-keys/0.log" Jul 15 15:22:10.570708 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:10.570682 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-345426-qjk7j_221f7ea2-c731-4112-82b5-1f2b6e6f64d3/revoke-keys/0.log" Jul 15 15:22:10.792020 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:10.791953 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-e47fbc-9qq6b_28aeefae-c8bd-469b-9dc0-cf4ab17f50ae/revoke-keys/0.log" Jul 15 15:22:11.232307 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:11.232278 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-ca6c4f33-x9qwv_8c327984-cfe6-46b9-9b35-1bfb416d8a9b/revoke-keys/0.log" Jul 15 15:22:11.348327 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:11.348303 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-0c81d7-ngthb_e0558d53-7e14-43a2-b621-6a61e0732bb8/revoke-keys/0.log" Jul 15 15:22:12.873456 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:12.873430 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d449b7fcf-9zf46_a89662fc-7a0b-40c9-8c3b-34483f6be84c/authorino/0.log" Jul 15 15:22:12.990667 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:12.990640 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-ts4lq_e5a03076-1c4b-4cde-b963-862cdb07c9cd/manager/0.log" Jul 15 15:22:13.361065 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:13.361033 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-s5b7f_a657d514-40bf-4944-b6b3-3dc8a2293643/registry-server/0.log" Jul 15 15:22:13.715495 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:13.715412 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ctjjr_c43ede6a-ac70-4059-aeb0-20513e15562e/manager/0.log" Jul 15 15:22:14.075545 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:14.075515 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mm47h_36f2caa4-a7a4-483d-98d1-03f471c890e9/istio-proxy/0.log" Jul 15 15:22:14.546862 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:14.546787 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-ng2vb_008f284e-86e4-435f-8ab2-086bce5953a9/istio-proxy/1.log" Jul 15 15:22:15.238548 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:15.238513 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-bzcll_2f419953-ed45-4208-bcb4-eece3355a9e3/main/0.log" Jul 15 15:22:15.582273 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:15.582237 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbrk2wg_ec4e88f3-533d-40f3-bf2f-0d7cde309ddd/main/0.log" Jul 15 15:22:15.809475 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:15.809447 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-5pxjj_89395a8a-a38d-4b7a-b059-4419aac9ca1d/main/0.log" Jul 15 15:22:24.052811 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:24.052780 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rgz47_e85e4589-15f8-4790-b90f-246692b35897/global-pull-secret-syncer/0.log" Jul 15 15:22:24.132636 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:24.132611 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5cdr7_ac545d6a-d123-455e-80d4-890435b906a7/konnectivity-agent/0.log" Jul 15 15:22:24.182290 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:24.182262 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-83.ec2.internal_eeee09bf5e958e41488ee1c3e17d2133/haproxy/0.log" Jul 15 15:22:29.334753 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:29.334718 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5d449b7fcf-9zf46_a89662fc-7a0b-40c9-8c3b-34483f6be84c/authorino/0.log" Jul 15 15:22:29.379498 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:29.379470 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-ts4lq_e5a03076-1c4b-4cde-b963-862cdb07c9cd/manager/0.log" Jul 15 15:22:29.515650 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:29.515623 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-s5b7f_a657d514-40bf-4944-b6b3-3dc8a2293643/registry-server/0.log" Jul 15 15:22:29.724018 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:29.723943 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ctjjr_c43ede6a-ac70-4059-aeb0-20513e15562e/manager/0.log" Jul 15 15:22:31.490230 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:31.490159 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_409a0c7c-4e42-4c35-b039-76669319eb94/alertmanager/0.log" Jul 15 15:22:31.513631 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:31.513606 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_409a0c7c-4e42-4c35-b039-76669319eb94/config-reloader/0.log" Jul 15 15:22:31.537901 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:31.537867 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_409a0c7c-4e42-4c35-b039-76669319eb94/kube-rbac-proxy-web/0.log" Jul 15 15:22:31.562386 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:31.562365 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_409a0c7c-4e42-4c35-b039-76669319eb94/kube-rbac-proxy/0.log" Jul 15 15:22:31.592545 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:31.592523 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_409a0c7c-4e42-4c35-b039-76669319eb94/kube-rbac-proxy-metric/0.log" Jul 15 15:22:31.620857 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:31.620841 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_409a0c7c-4e42-4c35-b039-76669319eb94/prom-label-proxy/0.log" Jul 15 15:22:31.647035 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:31.647014 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_409a0c7c-4e42-4c35-b039-76669319eb94/init-config-reloader/0.log" Jul 15 15:22:31.975177 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:31.975143 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qbdml_d6b362ea-1999-46e1-856a-06f61e53fb91/node-exporter/0.log" Jul 15 15:22:31.997380 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:31.997358 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qbdml_d6b362ea-1999-46e1-856a-06f61e53fb91/kube-rbac-proxy/0.log" Jul 15 15:22:32.023355 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:32.023337 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qbdml_d6b362ea-1999-46e1-856a-06f61e53fb91/init-textfile/0.log" Jul 15 15:22:32.137544 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:32.137523 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-slghk_7771d412-bf38-42fb-b668-f8bde1441f12/kube-rbac-proxy-main/0.log" Jul 15 15:22:32.160790 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:32.160772 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-slghk_7771d412-bf38-42fb-b668-f8bde1441f12/kube-rbac-proxy-self/0.log" Jul 15 15:22:32.182468 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:32.182446 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-slghk_7771d412-bf38-42fb-b668-f8bde1441f12/openshift-state-metrics/0.log" Jul 15 15:22:32.528342 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:32.528321 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65964b6d55-57llg_776ec54b-28e9-42b0-aa2f-5a6fa2e3e992/telemeter-client/0.log" Jul 15 15:22:32.551783 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:32.551758 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65964b6d55-57llg_776ec54b-28e9-42b0-aa2f-5a6fa2e3e992/reload/0.log" Jul 15 15:22:32.578117 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:32.578094 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65964b6d55-57llg_776ec54b-28e9-42b0-aa2f-5a6fa2e3e992/kube-rbac-proxy/0.log" Jul 15 15:22:32.607871 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:32.607850 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6668679d49-lzkx5_04d62702-a012-4138-85f4-0cf30f0c8aab/thanos-query/0.log" Jul 15 15:22:32.631452 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:32.631427 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6668679d49-lzkx5_04d62702-a012-4138-85f4-0cf30f0c8aab/kube-rbac-proxy-web/0.log" Jul 15 15:22:32.658530 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:32.658514 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6668679d49-lzkx5_04d62702-a012-4138-85f4-0cf30f0c8aab/kube-rbac-proxy/0.log" Jul 15 15:22:32.682011 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:32.681994 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6668679d49-lzkx5_04d62702-a012-4138-85f4-0cf30f0c8aab/prom-label-proxy/0.log" Jul 15 15:22:32.705395 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:32.705337 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6668679d49-lzkx5_04d62702-a012-4138-85f4-0cf30f0c8aab/kube-rbac-proxy-rules/0.log" Jul 15 15:22:32.728648 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:32.728626 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6668679d49-lzkx5_04d62702-a012-4138-85f4-0cf30f0c8aab/kube-rbac-proxy-metrics/0.log" Jul 15 15:22:35.101935 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.101907 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-5bgbj_996f65dd-80ac-421a-99c4-8a9dcfaf287b/download-server/0.log" Jul 15 15:22:35.553633 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.553603 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6"] Jul 15 15:22:35.553989 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.553975 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0558d53-7e14-43a2-b621-6a61e0732bb8" containerName="revoke-keys" Jul 15 15:22:35.554073 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.553991 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0558d53-7e14-43a2-b621-6a61e0732bb8" containerName="revoke-keys" Jul 15 15:22:35.554073 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.554070 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0558d53-7e14-43a2-b621-6a61e0732bb8" containerName="revoke-keys" Jul 15 15:22:35.557123 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.557108 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.559356 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.559336 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bqlw8\"/\"openshift-service-ca.crt\"" Jul 15 15:22:35.560611 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.560587 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bqlw8\"/\"default-dockercfg-dd5lq\"" Jul 15 15:22:35.560611 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.560596 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bqlw8\"/\"kube-root-ca.crt\"" Jul 15 15:22:35.565451 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.565430 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6"] Jul 15 15:22:35.654876 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.654846 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b84d2ef-7f4f-4745-9d0f-727954ec4c9b-lib-modules\") pod \"perf-node-gather-daemonset-znxw6\" (UID: \"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.655033 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.654905 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9b84d2ef-7f4f-4745-9d0f-727954ec4c9b-proc\") pod \"perf-node-gather-daemonset-znxw6\" (UID: \"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.655033 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.654934 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzn42\" (UniqueName: \"kubernetes.io/projected/9b84d2ef-7f4f-4745-9d0f-727954ec4c9b-kube-api-access-dzn42\") pod \"perf-node-gather-daemonset-znxw6\" (UID: \"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.655033 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.654965 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b84d2ef-7f4f-4745-9d0f-727954ec4c9b-sys\") pod \"perf-node-gather-daemonset-znxw6\" (UID: \"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.655033 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.655025 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9b84d2ef-7f4f-4745-9d0f-727954ec4c9b-podres\") pod \"perf-node-gather-daemonset-znxw6\" (UID: \"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.756317 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.756276 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9b84d2ef-7f4f-4745-9d0f-727954ec4c9b-podres\") pod \"perf-node-gather-daemonset-znxw6\" (UID: \"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.756504 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.756352 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b84d2ef-7f4f-4745-9d0f-727954ec4c9b-lib-modules\") pod \"perf-node-gather-daemonset-znxw6\" (UID: \"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.756504 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.756382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9b84d2ef-7f4f-4745-9d0f-727954ec4c9b-proc\") pod \"perf-node-gather-daemonset-znxw6\" (UID: \"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.756504 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.756401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzn42\" (UniqueName: \"kubernetes.io/projected/9b84d2ef-7f4f-4745-9d0f-727954ec4c9b-kube-api-access-dzn42\") pod \"perf-node-gather-daemonset-znxw6\" (UID: \"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.756504 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.756422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b84d2ef-7f4f-4745-9d0f-727954ec4c9b-sys\") pod \"perf-node-gather-daemonset-znxw6\" (UID: \"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.756504 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.756442 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9b84d2ef-7f4f-4745-9d0f-727954ec4c9b-podres\") pod \"perf-node-gather-daemonset-znxw6\" (UID: \"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.756504 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.756495 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9b84d2ef-7f4f-4745-9d0f-727954ec4c9b-proc\") pod \"perf-node-gather-daemonset-znxw6\" (UID: \"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.756735 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.756495 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b84d2ef-7f4f-4745-9d0f-727954ec4c9b-sys\") pod \"perf-node-gather-daemonset-znxw6\" (UID: \"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.756735 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.756530 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b84d2ef-7f4f-4745-9d0f-727954ec4c9b-lib-modules\") pod \"perf-node-gather-daemonset-znxw6\" (UID: \"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.764912 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.764874 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzn42\" (UniqueName: \"kubernetes.io/projected/9b84d2ef-7f4f-4745-9d0f-727954ec4c9b-kube-api-access-dzn42\") pod \"perf-node-gather-daemonset-znxw6\" (UID: \"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b\") " pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.867777 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.867692 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:35.997394 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:35.997369 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6"] Jul 15 15:22:35.999695 ip-10-0-139-83 kubenswrapper[2574]: W0715 15:22:35.999669 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9b84d2ef_7f4f_4745_9d0f_727954ec4c9b.slice/crio-2788d3d27adc8664473f7b9bfb77933334ea56477762b54afbf7e48cb05a5c76 WatchSource:0}: Error finding container 2788d3d27adc8664473f7b9bfb77933334ea56477762b54afbf7e48cb05a5c76: Status 404 returned error can't find the container with id 2788d3d27adc8664473f7b9bfb77933334ea56477762b54afbf7e48cb05a5c76 Jul 15 15:22:36.843976 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:36.843940 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" event={"ID":"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b","Type":"ContainerStarted","Data":"da045fd2f75c02fdc14375995701387477799e0f4ee86f530615783ebc626cb7"} Jul 15 15:22:36.844371 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:36.843991 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" event={"ID":"9b84d2ef-7f4f-4745-9d0f-727954ec4c9b","Type":"ContainerStarted","Data":"2788d3d27adc8664473f7b9bfb77933334ea56477762b54afbf7e48cb05a5c76"} Jul 15 15:22:36.844371 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:36.844020 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:36.863165 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:36.863120 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" podStartSLOduration=1.863104829 podStartE2EDuration="1.863104829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 15:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 15:22:36.861971776 +0000 UTC m=+2929.568047885" watchObservedRunningTime="2026-07-15 15:22:36.863104829 +0000 UTC m=+2929.569180934" Jul 15 15:22:36.871391 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:36.871369 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bcjgw_2aac9896-1d6f-476c-8a30-39ffb309a3de/dns/0.log" Jul 15 15:22:36.897433 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:36.897412 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bcjgw_2aac9896-1d6f-476c-8a30-39ffb309a3de/kube-rbac-proxy/0.log" Jul 15 15:22:37.022390 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:37.022357 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6sxqv_50bab32c-4edf-4ebe-9bd0-cfa9ef0df269/dns-node-resolver/0.log" Jul 15 15:22:37.652468 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:37.652439 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-djz58_5c59c38d-2211-4547-80df-7505bb4ef76d/node-ca/0.log" Jul 15 15:22:38.717895 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:38.717855 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mm47h_36f2caa4-a7a4-483d-98d1-03f471c890e9/istio-proxy/0.log" Jul 15 15:22:38.919695 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:38.919666 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-ng2vb_008f284e-86e4-435f-8ab2-086bce5953a9/istio-proxy/0.log" Jul 15 15:22:38.948103 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:38.948075 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-ng2vb_008f284e-86e4-435f-8ab2-086bce5953a9/istio-proxy/1.log" Jul 15 15:22:39.770959 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:39.770934 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4fmsb_3cbfdfaf-86e4-4a16-83f5-fa3cc2e19ac5/serve-healthcheck-canary/0.log" Jul 15 15:22:40.434699 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:40.434675 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lzntz_d5454a5b-2852-4d93-b072-4d3b7756974d/kube-rbac-proxy/0.log" Jul 15 15:22:40.460577 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:40.460549 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lzntz_d5454a5b-2852-4d93-b072-4d3b7756974d/exporter/0.log" Jul 15 15:22:40.483333 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:40.483314 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lzntz_d5454a5b-2852-4d93-b072-4d3b7756974d/extractor/0.log" Jul 15 15:22:42.858238 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:42.858210 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bqlw8/perf-node-gather-daemonset-znxw6" Jul 15 15:22:43.234048 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:43.233958 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr_90bc98c4-0476-47c2-a934-24ec14e2be01/manager/0.log" Jul 15 15:22:43.254920 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:43.254874 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6dcffcc7-nvdqr_90bc98c4-0476-47c2-a934-24ec14e2be01/copy-manifests/0.log" Jul 15 15:22:44.354639 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:44.354607 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mm47h_36f2caa4-a7a4-483d-98d1-03f471c890e9/istio-proxy/0.log" Jul 15 15:22:44.531946 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:44.531925 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-ng2vb_008f284e-86e4-435f-8ab2-086bce5953a9/istio-proxy/0.log" Jul 15 15:22:44.560228 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:44.560203 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-ng2vb_008f284e-86e4-435f-8ab2-086bce5953a9/istio-proxy/1.log" Jul 15 15:22:44.584406 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:44.584386 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68975544d6-jldvj_efe244d8-1cf0-488d-ac8b-ae6847f2f1bc/manager/0.log" Jul 15 15:22:45.975973 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:45.975949 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mm47h_36f2caa4-a7a4-483d-98d1-03f471c890e9/istio-proxy/0.log" Jul 15 15:22:46.131008 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:46.130979 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-ng2vb_008f284e-86e4-435f-8ab2-086bce5953a9/istio-proxy/0.log" Jul 15 15:22:46.160626 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:46.160591 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-ng2vb_008f284e-86e4-435f-8ab2-086bce5953a9/istio-proxy/1.log" Jul 15 15:22:51.422313 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:51.422277 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4hzj7_0ea1b539-6ec9-41b2-91aa-c27772f97451/kube-multus/0.log" Jul 15 15:22:51.512645 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:51.512581 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cd8gf_960e795d-fb41-4f9d-8b1f-533f625dbb08/kube-multus-additional-cni-plugins/0.log" Jul 15 15:22:51.534867 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:51.534842 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cd8gf_960e795d-fb41-4f9d-8b1f-533f625dbb08/egress-router-binary-copy/0.log" Jul 15 15:22:51.556693 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:51.556671 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cd8gf_960e795d-fb41-4f9d-8b1f-533f625dbb08/cni-plugins/0.log" Jul 15 15:22:51.588236 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:51.588215 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cd8gf_960e795d-fb41-4f9d-8b1f-533f625dbb08/bond-cni-plugin/0.log" Jul 15 15:22:51.611350 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:51.611331 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cd8gf_960e795d-fb41-4f9d-8b1f-533f625dbb08/routeoverride-cni/0.log" Jul 15 15:22:51.633430 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:51.633407 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cd8gf_960e795d-fb41-4f9d-8b1f-533f625dbb08/whereabouts-cni-bincopy/0.log" Jul 15 15:22:51.657272 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:51.657253 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cd8gf_960e795d-fb41-4f9d-8b1f-533f625dbb08/whereabouts-cni/0.log" Jul 15 15:22:52.050022 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:52.049994 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9jbvd_7d23262f-6be8-46f4-9122-28ce9683ed4a/network-metrics-daemon/0.log" Jul 15 15:22:52.074192 ip-10-0-139-83 kubenswrapper[2574]: I0715 15:22:52.074172 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9jbvd_7d23262f-6be8-46f4-9122-28ce9683ed4a/kube-rbac-proxy/0.log"