Jul 14 19:44:22.658945 ip-10-0-132-184 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 14 19:44:22.658962 ip-10-0-132-184 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 14 19:44:22.658972 ip-10-0-132-184 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 14 19:44:22.659316 ip-10-0-132-184 systemd[1]: Failed to start Kubernetes Kubelet. Jul 14 19:44:32.741737 ip-10-0-132-184 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 14 19:44:32.741754 ip-10-0-132-184 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d3514be83d554eacb06afcdc3234a4c5 -- Jul 14 19:46:20.740388 ip-10-0-132-184 systemd[1]: Starting Kubernetes Kubelet... Jul 14 19:46:21.098872 ip-10-0-132-184 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:46:21.098872 ip-10-0-132-184 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 19:46:21.098872 ip-10-0-132-184 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:46:21.098872 ip-10-0-132-184 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 19:46:21.098872 ip-10-0-132-184 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:46:21.100430 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.100316 2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 19:46:21.104829 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104814 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:46:21.104829 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104829 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104833 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104836 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104839 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104843 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104846 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104849 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104852 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104855 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104858 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104866 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104869 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104872 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104874 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104877 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104880 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104882 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104885 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104889 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:46:21.104889 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104893 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104896 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104899 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104902 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104905 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104908 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104911 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104913 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104916 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104918 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104920 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104923 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104925 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104928 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104931 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104933 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104936 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104938 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104941 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:46:21.105350 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104943 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104948 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104951 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104954 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104957 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104960 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104963 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104966 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104969 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104971 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104973 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104976 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104978 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104981 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104983 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104987 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104990 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104992 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104995 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.104997 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:46:21.105818 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105000 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105003 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105005 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105008 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105010 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105013 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105015 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105018 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105020 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105023 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105025 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105028 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105030 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105033 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105035 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105038 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105040 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105046 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105050 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:46:21.106300 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105053 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105056 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105058 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105061 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105064 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105067 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105069 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105072 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105075 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105078 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105081 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105083 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105086 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105088 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105091 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105093 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105095 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105098 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105101 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105103 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:46:21.106750 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105118 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105804 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105810 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105813 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105816 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105819 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105822 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105825 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105828 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105831 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105834 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105836 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105839 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105842 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105845 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105847 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105850 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105852 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105857 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105862 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:46:21.107269 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105865 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105868 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105871 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105873 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105876 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105878 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105881 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105884 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105886 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105889 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105892 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105895 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105897 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105900 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105904 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105907 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105910 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105913 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105916 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:46:21.107749 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105919 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105923 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105925 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105928 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105931 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105933 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105936 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105939 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105942 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105944 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105946 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105949 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105952 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105955 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105957 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105960 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105962 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105965 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105967 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105969 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:46:21.108253 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105972 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105975 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105977 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105979 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105982 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105985 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105987 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105990 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105992 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105995 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.105997 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106000 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106002 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106005 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106007 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106010 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106012 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106015 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106018 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:46:21.108744 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106022 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106025 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106028 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106031 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106033 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106037 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106040 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106043 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106045 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106048 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106051 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106053 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106056 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106059 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106061 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106063 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106066 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106068 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106071 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106073 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:46:21.109213 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106076 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106078 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106460 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106465 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106468 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106471 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106473 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106476 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106479 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106481 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106484 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106486 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106489 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106492 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106495 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106497 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106501 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106504 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106506 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106509 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:46:21.109682 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106511 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106522 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106525 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106527 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106530 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106532 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106535 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106537 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106540 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106542 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106545 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106547 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106550 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106553 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106555 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106557 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106560 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106563 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106565 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106567 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:46:21.110185 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106572 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106576 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106579 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106581 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106583 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106587 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106589 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106592 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106595 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106598 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106601 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106603 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106605 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106608 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106610 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106613 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106615 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106617 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106620 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106622 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:46:21.110656 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106625 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106627 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106630 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106632 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106636 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106639 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106642 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106645 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106648 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106651 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106654 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106658 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106661 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106664 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106666 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106669 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106672 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106675 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106678 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:46:21.111141 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106681 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106684 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106687 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106690 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106692 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106695 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106697 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106700 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106702 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106705 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106707 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106709 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106712 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106714 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106717 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106719 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106722 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106724 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106727 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106729 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:46:21.111588 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106732 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.106734 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106809 2557 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106816 2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106823 2557 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106827 2557 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106831 2557 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106836 2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106841 2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106845 2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106848 2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106852 2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106856 2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106859 2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106864 2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106866 2557 flags.go:64] FLAG: --cgroup-root="" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106869 2557 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106872 2557 flags.go:64] FLAG: --client-ca-file="" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106875 2557 flags.go:64] FLAG: --cloud-provider="external" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106878 2557 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106882 2557 flags.go:64] FLAG: --cluster-domain="" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106885 2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106888 2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106891 2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 19:46:21.112057 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106894 2557 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106898 2557 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106901 2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106904 2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106908 2557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106911 2557 flags.go:64] FLAG: --contention-profiling="false" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106913 2557 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106916 2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106919 2557 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106922 2557 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106930 2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106933 2557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106936 2557 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106939 2557 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106942 2557 flags.go:64] FLAG: --enable-server="true" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106946 2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106951 2557 flags.go:64] FLAG: --event-burst="100" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106954 2557 flags.go:64] FLAG: --event-qps="50" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106957 2557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106960 2557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106963 2557 flags.go:64] FLAG: --eviction-hard="" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106968 2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106971 2557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106973 2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106976 2557 flags.go:64] FLAG: --eviction-soft="" Jul 14 19:46:21.112623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106979 2557 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106982 2557 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106985 2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106988 2557 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106991 2557 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106993 2557 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.106996 2557 flags.go:64] FLAG: --feature-gates="" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107000 2557 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107003 2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107006 2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107009 2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107012 2557 flags.go:64] FLAG: --healthz-port="10248" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107015 2557 flags.go:64] FLAG: --help="false" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107018 2557 flags.go:64] FLAG: --hostname-override="ip-10-0-132-184.ec2.internal" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107021 2557 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107024 2557 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107027 2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107032 2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107036 2557 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107038 2557 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107041 2557 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107044 2557 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107048 2557 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 19:46:21.113258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107051 2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107054 2557 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107057 2557 flags.go:64] FLAG: --kube-reserved="" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107060 2557 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107063 2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107066 2557 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107069 2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107072 2557 flags.go:64] FLAG: --lock-file="" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107075 2557 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107077 2557 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107080 2557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107086 2557 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107088 2557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107091 2557 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107094 2557 flags.go:64] FLAG: --logging-format="text" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107096 2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107099 2557 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107102 2557 flags.go:64] FLAG: --manifest-url="" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107120 2557 flags.go:64] FLAG: --manifest-url-header="" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107124 2557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107128 2557 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107132 2557 flags.go:64] FLAG: --max-pods="110" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107135 2557 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107138 2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107140 2557 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 19:46:21.113860 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107143 2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107148 2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107150 2557 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107153 2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107160 2557 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107163 2557 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107168 2557 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107171 2557 flags.go:64] FLAG: --pod-cidr="" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107174 2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107180 2557 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107183 2557 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107186 2557 flags.go:64] FLAG: --pods-per-core="0" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107189 2557 flags.go:64] FLAG: --port="10250" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107192 2557 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107196 2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05a460de9a6945601" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107199 2557 flags.go:64] FLAG: --qos-reserved="" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107202 2557 flags.go:64] FLAG: --read-only-port="10255" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107205 2557 flags.go:64] FLAG: --register-node="true" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107208 2557 flags.go:64] FLAG: --register-with-taints="" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107212 2557 flags.go:64] FLAG: --registry-burst="10" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107214 2557 flags.go:64] FLAG: --registry-qps="5" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107217 2557 flags.go:64] FLAG: --reserved-cpus="" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107220 2557 flags.go:64] FLAG: --reserved-memory="" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107224 2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107227 2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 19:46:21.114465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107230 2557 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107233 2557 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107236 2557 flags.go:64] FLAG: --runonce="false" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107239 2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107242 2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107245 2557 flags.go:64] FLAG: --seccomp-default="false" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107248 2557 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107250 2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107255 2557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107258 2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107261 2557 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107264 2557 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107267 2557 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107270 2557 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107274 2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107278 2557 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107281 2557 flags.go:64] FLAG: --system-cgroups="" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107284 2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107289 2557 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107292 2557 flags.go:64] FLAG: --tls-cert-file="" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107295 2557 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107299 2557 flags.go:64] FLAG: --tls-min-version="" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107302 2557 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107305 2557 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107308 2557 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 19:46:21.115055 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107310 2557 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107313 2557 flags.go:64] FLAG: --v="2" Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107317 2557 flags.go:64] FLAG: --version="false" Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107321 2557 flags.go:64] FLAG: --vmodule="" Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107325 2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.107328 2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107442 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107446 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107449 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107452 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107454 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107457 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107460 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107463 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107465 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107469 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107472 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107474 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107477 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107480 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107482 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:46:21.115676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107488 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107491 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107493 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107496 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107498 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107501 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107503 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107506 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107509 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107512 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107514 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107517 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107519 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107522 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107524 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107527 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107529 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107532 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107535 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107537 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:46:21.116193 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107539 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107542 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107544 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107547 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107549 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107552 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107557 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107560 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107563 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107566 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107568 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107571 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107575 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107578 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107580 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107583 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107585 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107588 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107591 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:46:21.116694 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107593 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107597 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107601 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107604 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107607 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107610 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107613 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107615 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107618 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107621 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107624 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107626 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107629 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107631 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107633 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107636 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107638 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107641 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107643 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:46:21.117163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107647 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107650 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107653 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107655 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107659 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107662 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107666 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107669 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107672 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107675 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107677 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107680 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107682 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107685 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107688 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107690 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107692 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107695 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107698 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:46:21.117606 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107701 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:46:21.118066 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107703 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:46:21.118066 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107706 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:46:21.118066 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107729 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:46:21.118066 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107733 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:46:21.118066 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107735 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:46:21.118066 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.107739 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:46:21.118066 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.108252 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:46:21.118066 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.114760 2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 19:46:21.118066 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.114774 2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 19:46:21.118066 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.114790 2557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 19:46:21.118066 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.114799 2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 19:46:21.118066 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114851 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:46:21.118066 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114856 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:46:21.118066 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114859 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:46:21.118066 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114862 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114865 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114868 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114871 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114873 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114876 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114879 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114882 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114884 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114887 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114891 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114895 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114898 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114901 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114904 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114907 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114910 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114912 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114915 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:46:21.118443 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114918 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114920 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114922 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114926 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114928 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114931 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114934 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114937 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114940 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114942 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114945 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114948 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114950 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114953 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114956 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114958 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114961 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114963 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114966 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114969 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114971 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:46:21.118896 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114974 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114977 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114979 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114981 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114984 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114987 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114990 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114992 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114995 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.114998 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115001 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115003 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115007 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115010 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115012 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115015 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115018 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115021 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115024 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:46:21.119407 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115026 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115029 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115032 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115034 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115038 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115041 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115044 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115047 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115050 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115052 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115056 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115058 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115061 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115064 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115066 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115069 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115072 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115075 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115078 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:46:21.119866 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115081 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115083 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115086 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115089 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115091 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115094 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115096 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115099 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115101 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115104 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115121 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115124 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115127 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115129 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115132 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115135 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115137 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:46:21.120324 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115140 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:46:21.120741 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.115145 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:46:21.120741 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115258 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:46:21.120741 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115262 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:46:21.120741 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115265 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:46:21.120741 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115268 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:46:21.120741 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115270 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:46:21.120741 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115273 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:46:21.120741 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115276 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:46:21.120741 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115279 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:46:21.120741 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115282 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:46:21.120741 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115284 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:46:21.120741 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115287 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:46:21.120741 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115289 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:46:21.120741 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115293 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:46:21.120741 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115295 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115298 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115302 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115306 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115309 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115312 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115314 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115318 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115320 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115323 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115326 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115329 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115332 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115335 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115338 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115340 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115344 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115346 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115350 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115353 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:46:21.121130 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115355 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115358 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115360 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115363 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115365 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115368 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115370 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115372 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115375 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115377 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115380 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115382 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115385 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115387 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115390 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115392 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115395 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115397 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115400 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115402 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:46:21.121613 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115405 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115408 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115410 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115413 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115416 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115419 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115422 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115425 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115428 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115430 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115433 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115435 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115438 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115440 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115443 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115445 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115448 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115450 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115453 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:46:21.122074 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115455 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115458 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115460 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115462 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115465 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115468 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115470 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115473 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115475 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115478 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115480 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115483 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115485 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115488 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115490 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115493 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115496 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115499 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115503 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115505 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:46:21.122536 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115508 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:46:21.123009 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115510 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:46:21.123009 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115512 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:46:21.123009 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115515 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:46:21.123009 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115517 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:46:21.123009 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115520 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:46:21.123009 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:21.115522 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:46:21.123009 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.115527 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:46:21.123009 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.115625 2557 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 19:46:21.123009 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.117510 2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 19:46:21.123009 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.118716 2557 server.go:1019] "Starting client certificate rotation" Jul 14 19:46:21.123009 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.118811 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 19:46:21.123009 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.119518 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 19:46:21.123009 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.121674 2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 19:46:21.125906 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.125881 2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 19:46:21.133556 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.133538 2557 log.go:25] "Validated CRI v1 runtime API" Jul 14 19:46:21.137593 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.137578 2557 log.go:25] "Validated CRI v1 image API" Jul 14 19:46:21.138490 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.138474 2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 19:46:21.140596 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.140570 2557 fs.go:135] Filesystem UUIDs: map[0bc0c434-4bed-4718-9bee-bc0eb8c5e6f7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d8086387-27ae-43b3-ad43-d168d92bceda:/dev/nvme0n1p3] Jul 14 19:46:21.140660 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.140595 2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 19:46:21.144684 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.144581 2557 manager.go:216] Machine: {Timestamp:2026-07-14 19:46:21.144103567 +0000 UTC m=+0.298796651 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099498 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec200a06e883dd7859d526f9165d73de SystemUUID:ec200a06-e883-dd78-59d5-26f9165d73de BootID:d3514be8-3d55-4eac-b06a-fcdc3234a4c5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e5:33:1b:b4:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e5:33:1b:b4:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:23:54:4a:81:91 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 19:46:21.144684 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.144681 2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 19:46:21.144799 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.144752 2557 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 19:46:21.146262 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.146235 2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 19:46:21.146390 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.146265 2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-184.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 19:46:21.146936 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.146926 2557 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 19:46:21.146976 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.146938 2557 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 19:46:21.146976 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.146946 2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 19:46:21.147694 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.147681 2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 19:46:21.148938 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.148928 2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 19:46:21.150532 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.150512 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 19:46:21.152033 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.152022 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 14 19:46:21.152161 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.152152 2557 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 19:46:21.152199 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.152180 2557 kubelet.go:486] "Attempting to sync node with API server" Jul 14 19:46:21.152199 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.152190 2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 19:46:21.152255 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.152203 2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 19:46:21.152255 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.152213 2557 kubelet.go:398] "Adding apiserver pod source" Jul 14 19:46:21.152255 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.152221 2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 19:46:21.152574 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.152557 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 19:46:21.152618 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.152576 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 19:46:21.155418 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.155403 2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 19:46:21.159217 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.159200 2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 19:46:21.159280 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.159232 2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 19:46:21.159396 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.159383 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 19:46:21.159428 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.159398 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 19:46:21.159428 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.159404 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 19:46:21.159428 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.159409 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 19:46:21.159428 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.159417 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 19:46:21.159428 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.159422 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 19:46:21.160014 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.160005 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 19:46:21.160014 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.160014 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 19:46:21.160066 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.160020 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 19:46:21.160066 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.160025 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 19:46:21.160066 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.160034 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 19:46:21.160066 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.160042 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 19:46:21.160757 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.160747 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 19:46:21.160757 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.160757 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 19:46:21.163820 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.163805 2557 server.go:1268] "Started kubelet" Jul 14 19:46:21.163941 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.163897 2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 19:46:21.163998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.163967 2557 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 19:46:21.164187 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.164132 2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 19:46:21.164619 ip-10-0-132-184 systemd[1]: Started Kubernetes Kubelet. Jul 14 19:46:21.164972 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:21.164949 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 19:46:21.165051 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:21.165008 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-184.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 19:46:21.165051 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.165045 2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-184.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 19:46:21.165231 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.165096 2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 19:46:21.166310 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.166296 2557 server.go:310] "Adding debug handlers to kubelet server" Jul 14 19:46:21.167992 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:21.167974 2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 19:46:21.171272 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.171255 2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 19:46:21.171353 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.171272 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 19:46:21.171412 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:21.171395 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-184.ec2.internal\" not found" Jul 14 19:46:21.171906 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.171867 2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 19:46:21.171906 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.171882 2557 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 19:46:21.171906 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.171896 2557 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 19:46:21.172084 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.172001 2557 reconstruct.go:97] "Volume reconstruction finished" Jul 14 19:46:21.172084 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.172010 2557 reconciler.go:29] "Reconciler: start to sync state" Jul 14 19:46:21.172202 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.172088 2557 factory.go:55] Registering systemd factory Jul 14 19:46:21.172202 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.172104 2557 factory.go:223] Registration of the systemd container factory successfully Jul 14 19:46:21.172426 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.172383 2557 factory.go:153] Registering CRI-O factory Jul 14 19:46:21.172426 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.172402 2557 factory.go:223] Registration of the crio container factory successfully Jul 14 19:46:21.172575 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.172450 2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 19:46:21.172575 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.172473 2557 factory.go:103] Registering Raw factory Jul 14 19:46:21.172575 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.172488 2557 manager.go:1196] Started watching for new ooms in manager Jul 14 19:46:21.172994 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:21.172209 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-184.ec2.internal.18c23fa533d34f6a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-184.ec2.internal,UID:ip-10-0-132-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-184.ec2.internal,},FirstTimestamp:2026-07-14 19:46:21.163777898 +0000 UTC m=+0.318470984,LastTimestamp:2026-07-14 19:46:21.163777898 +0000 UTC m=+0.318470984,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-184.ec2.internal,}" Jul 14 19:46:21.172994 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.172987 2557 manager.go:349] Starting recovery of all containers Jul 14 19:46:21.178963 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:21.178932 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 19:46:21.179133 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:21.179095 2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-184.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 19:46:21.183935 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.183322 2557 manager.go:354] Recovery completed Jul 14 19:46:21.186008 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:21.185978 2557 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 14 19:46:21.188943 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.188931 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:46:21.191493 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.191463 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:46:21.191571 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.191514 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:46:21.191571 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.191529 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:46:21.191652 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.191576 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6zrx8" Jul 14 19:46:21.192049 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.192035 2557 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 19:46:21.192139 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.192050 2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 19:46:21.192139 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.192068 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 14 19:46:21.192139 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.192050 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:46:21.192861 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.192843 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:46:21.192930 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.192872 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:46:21.192930 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.192881 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:46:21.193370 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.193348 2557 policy_none.go:49] "None policy: Start" Jul 14 19:46:21.193370 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.193368 2557 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 19:46:21.193479 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.193378 2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 19:46:21.193479 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:21.193404 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-184.ec2.internal.18c23fa5357a3a69 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-184.ec2.internal,UID:ip-10-0-132-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-184.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-184.ec2.internal,},FirstTimestamp:2026-07-14 19:46:21.191494249 +0000 UTC m=+0.346187336,LastTimestamp:2026-07-14 19:46:21.191494249 +0000 UTC m=+0.346187336,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-184.ec2.internal,}" Jul 14 19:46:21.194362 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.194352 2557 policy_none.go:47] "Start" Jul 14 19:46:21.200409 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.200384 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6zrx8" Jul 14 19:46:21.231715 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.231698 2557 manager.go:337] "Starting Device Plugin manager" Jul 14 19:46:21.231826 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:21.231737 2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 19:46:21.231826 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.231750 2557 server.go:87] "Starting device plugin registration server" Jul 14 19:46:21.232009 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.231997 2557 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 19:46:21.232066 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.232011 2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 19:46:21.232131 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.232087 2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 19:46:21.232196 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.232185 2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 19:46:21.232196 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.232193 2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 19:46:21.232656 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:21.232629 2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 19:46:21.232756 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:21.232674 2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-184.ec2.internal\" not found" Jul 14 19:46:21.278231 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.278198 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 19:46:21.279432 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.279415 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 19:46:21.279499 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.279454 2557 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 19:46:21.279499 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:21.279477 2557 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 19:46:21.279593 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:21.279510 2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 19:46:22.357043 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.356839 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:22.408052 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.407998 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:46:22.409174 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.409158 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:46:22.409246 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.409190 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:46:22.409246 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.409205 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:46:22.409246 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.409237 2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.418506 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.418490 2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.418565 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:22.418515 2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-184.ec2.internal\": node \"ip-10-0-132-184.ec2.internal\" not found" Jul 14 19:46:22.432889 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:22.432840 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-184.ec2.internal\" not found" Jul 14 19:46:22.454910 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.454885 2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-184.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal"] Jul 14 19:46:22.454995 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.454950 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:46:22.455798 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.455783 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:46:22.455854 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.455809 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:46:22.455854 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.455819 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:46:22.457312 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.457300 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:46:22.457420 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.457405 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.457472 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.457436 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:46:22.458127 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.458113 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:46:22.458188 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.458140 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:46:22.458188 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.458154 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:46:22.458188 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.458116 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:46:22.458327 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.458203 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:46:22.458327 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.458222 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:46:22.459444 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.459427 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.459532 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.459458 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:46:22.460119 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.460104 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:46:22.460208 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.460132 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:46:22.460208 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.460144 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:46:22.481796 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:22.481777 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-184.ec2.internal\" not found" node="ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.486030 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:22.486012 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-184.ec2.internal\" not found" node="ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.533355 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:22.533333 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-184.ec2.internal\" not found" Jul 14 19:46:22.548166 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.548145 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/130523dbb878b2a1af0eb57e85edc42c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-184.ec2.internal\" (UID: \"130523dbb878b2a1af0eb57e85edc42c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.548245 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.548170 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4baf446df5fec806f69a038fd9e73461-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal\" (UID: \"4baf446df5fec806f69a038fd9e73461\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.548245 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.548188 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4baf446df5fec806f69a038fd9e73461-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal\" (UID: \"4baf446df5fec806f69a038fd9e73461\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.633668 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:22.633631 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-184.ec2.internal\" not found" Jul 14 19:46:22.649013 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.648984 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/130523dbb878b2a1af0eb57e85edc42c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-184.ec2.internal\" (UID: \"130523dbb878b2a1af0eb57e85edc42c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.649091 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.649040 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4baf446df5fec806f69a038fd9e73461-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal\" (UID: \"4baf446df5fec806f69a038fd9e73461\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.649091 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.649074 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4baf446df5fec806f69a038fd9e73461-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal\" (UID: \"4baf446df5fec806f69a038fd9e73461\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.649091 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.649000 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/130523dbb878b2a1af0eb57e85edc42c-config\") pod \"kube-apiserver-proxy-ip-10-0-132-184.ec2.internal\" (UID: \"130523dbb878b2a1af0eb57e85edc42c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.649200 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.649109 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4baf446df5fec806f69a038fd9e73461-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal\" (UID: \"4baf446df5fec806f69a038fd9e73461\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.649200 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.649124 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4baf446df5fec806f69a038fd9e73461-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal\" (UID: \"4baf446df5fec806f69a038fd9e73461\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.734453 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:22.734402 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-184.ec2.internal\" not found" Jul 14 19:46:22.783977 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.783951 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.788842 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:22.788827 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal" Jul 14 19:46:22.834626 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:22.834594 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-184.ec2.internal\" not found" Jul 14 19:46:22.935223 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:22.935172 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-184.ec2.internal\" not found" Jul 14 19:46:23.035722 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.035628 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-184.ec2.internal\" not found" Jul 14 19:46:23.136305 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.136260 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-184.ec2.internal\" not found" Jul 14 19:46:23.176396 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.176370 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:23.193534 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.193510 2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 19:46:23.193654 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.193631 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 19:46:23.193691 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.193672 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 19:46:23.227805 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.227780 2557 apiserver.go:52] "Watching apiserver" Jul 14 19:46:23.246707 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.246679 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 19:46:23.246866 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.246682 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.ec2.internal" Jul 14 19:46:23.246930 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.246683 2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 19:46:23.248477 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.248455 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-rqfrv","openshift-multus/network-metrics-daemon-bh66q","openshift-network-diagnostics/network-check-target-xppkv","openshift-network-operator/iptables-alerter-ck7z5","kube-system/konnectivity-agent-ff7s5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r","openshift-dns/node-resolver-lpvwr","openshift-ovn-kubernetes/ovnkube-node-926lj","kube-system/global-pull-secret-syncer-6w8gx","openshift-cluster-node-tuning-operator/tuned-d7bzh","openshift-image-registry/node-ca-knrrs","openshift-multus/multus-additional-cni-plugins-pdjn9"] Jul 14 19:46:23.251217 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.251191 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.252349 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.252311 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:23.252453 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.252408 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:23.253616 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.253594 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:23.253743 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.253605 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 19:46:23.253905 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.253887 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:23.253997 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.253966 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 19:46:23.253997 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.253956 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:23.254116 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.254043 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 19:46:23.254116 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.254079 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tzmln\"" Jul 14 19:46:23.254116 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.254051 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 19:46:23.254875 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.254860 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ck7z5" Jul 14 19:46:23.256018 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.256001 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ff7s5" Jul 14 19:46:23.256688 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.256672 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:46:23.257158 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.257133 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 19:46:23.257263 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.257178 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 19:46:23.257263 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.257212 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x49s5\"" Jul 14 19:46:23.257435 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.257422 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.258017 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.258000 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qjvdn\"" Jul 14 19:46:23.258155 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.258142 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 19:46:23.258246 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.258227 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 19:46:23.258473 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.258459 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lpvwr" Jul 14 19:46:23.259665 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.259650 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 19:46:23.259754 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.259705 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 19:46:23.259754 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.259717 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g4tpb\"" Jul 14 19:46:23.259872 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.259804 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 19:46:23.260049 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.260035 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.260604 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.260589 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 19:46:23.261018 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.261003 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 19:46:23.261119 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.261031 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wsd65\"" Jul 14 19:46:23.261322 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.261308 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:23.261394 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.261376 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:23.262332 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.262318 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4xc9b\"" Jul 14 19:46:23.262332 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.262325 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 19:46:23.262464 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.262375 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal" Jul 14 19:46:23.262464 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.262427 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 19:46:23.262556 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.262433 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 19:46:23.262723 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.262708 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 19:46:23.262794 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.262716 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 19:46:23.262853 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.262826 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 19:46:23.262960 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.262945 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.263029 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.262970 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 19:46:23.264725 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.264706 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-knrrs" Jul 14 19:46:23.265105 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.265090 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:46:23.265400 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.265385 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7dnwd\"" Jul 14 19:46:23.265489 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.265473 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 19:46:23.265901 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.265883 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-184.ec2.internal"] Jul 14 19:46:23.265997 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.265984 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.266772 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.266756 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 19:46:23.266972 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.266955 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 19:46:23.266972 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.266961 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 19:46:23.267151 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.266991 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 19:46:23.267289 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.267276 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5nfkz\"" Jul 14 19:46:23.267897 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.267881 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 19:46:23.267981 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.267915 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4c6r7\"" Jul 14 19:46:23.267981 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.267931 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 19:46:23.277786 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.277757 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 19:41:22 +0000 UTC" deadline="2028-03-10 13:04:08.747099923 +0000 UTC" Jul 14 19:46:23.277786 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.277785 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14513h17m45.469317085s" Jul 14 19:46:23.278052 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.278040 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 19:46:23.278108 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.278076 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal"] Jul 14 19:46:23.293467 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.293444 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g4xhz" Jul 14 19:46:23.304361 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.304338 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g4xhz" Jul 14 19:46:23.305544 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:23.305513 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130523dbb878b2a1af0eb57e85edc42c.slice/crio-406a88a6269e0b2846b9f4930dd34140fe224b65284b2b37a84fde98e6289a79 WatchSource:0}: Error finding container 406a88a6269e0b2846b9f4930dd34140fe224b65284b2b37a84fde98e6289a79: Status 404 returned error can't find the container with id 406a88a6269e0b2846b9f4930dd34140fe224b65284b2b37a84fde98e6289a79 Jul 14 19:46:23.305730 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:23.305713 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4baf446df5fec806f69a038fd9e73461.slice/crio-a7505e2c8bfec73ab7a91c0d739c3a5ef0f3c5d9e1415ef17b9eb5d68f047ef2 WatchSource:0}: Error finding container a7505e2c8bfec73ab7a91c0d739c3a5ef0f3c5d9e1415ef17b9eb5d68f047ef2: Status 404 returned error can't find the container with id a7505e2c8bfec73ab7a91c0d739c3a5ef0f3c5d9e1415ef17b9eb5d68f047ef2 Jul 14 19:46:23.310073 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.310046 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:46:23.348006 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.347990 2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 19:46:23.352531 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352513 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-os-release\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.352597 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352538 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-var-lib-cni-bin\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.352597 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352565 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15d09c81-aa45-4b25-83c2-c13f788e021c-ovnkube-script-lib\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.352667 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352607 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-host\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.352667 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352631 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/87dcc8db-8293-45b1-bb67-1db76191f899-etc-tuned\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.352667 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352646 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/839de40a-5fee-44af-8173-a65990f845cf-serviceca\") pod \"node-ca-knrrs\" (UID: \"839de40a-5fee-44af-8173-a65990f845cf\") " pod="openshift-image-registry/node-ca-knrrs" Jul 14 19:46:23.352667 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352661 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-etc-openvswitch\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.352777 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352674 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-lib-modules\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.352777 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352699 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87dcc8db-8293-45b1-bb67-1db76191f899-tmp\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.352777 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352726 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3f090a3-021a-4185-bcc8-7c00d7d56b77-kubelet-config\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:23.352777 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352744 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-cni-binary-copy\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.352777 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352762 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-cni-bin\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.352777 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352776 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-modprobe-d\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.353015 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352795 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-socket-dir\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.353015 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352809 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-kubernetes\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.353015 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352824 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c736c89e-7b1b-4d96-b36e-60956aca58af-system-cni-dir\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.353015 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352838 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdkvq\" (UniqueName: \"kubernetes.io/projected/c736c89e-7b1b-4d96-b36e-60956aca58af-kube-api-access-tdkvq\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.353015 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352879 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pddkq\" (UniqueName: \"kubernetes.io/projected/fd7db434-46fc-4a27-83b2-acaefc0bd727-kube-api-access-pddkq\") pod \"node-resolver-lpvwr\" (UID: \"fd7db434-46fc-4a27-83b2-acaefc0bd727\") " pod="openshift-dns/node-resolver-lpvwr" Jul 14 19:46:23.353015 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352915 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-run-k8s-cni-cncf-io\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.353015 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352938 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-run-netns\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.353015 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352964 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-multus-conf-dir\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.353015 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.352986 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-multus-daemon-config\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.353015 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353014 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h7cp\" (UniqueName: \"kubernetes.io/projected/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-kube-api-access-5h7cp\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353029 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-var-lib-openvswitch\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353043 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15d09c81-aa45-4b25-83c2-c13f788e021c-ovn-node-metrics-cert\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353071 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378-host-slash\") pod \"iptables-alerter-ck7z5\" (UID: \"cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378\") " pod="openshift-network-operator/iptables-alerter-ck7z5" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353085 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/322d4ffe-4783-4a5a-a18f-47878e117000-agent-certs\") pod \"konnectivity-agent-ff7s5\" (UID: \"322d4ffe-4783-4a5a-a18f-47878e117000\") " pod="kube-system/konnectivity-agent-ff7s5" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353106 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-systemd\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353135 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353157 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353181 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd7db434-46fc-4a27-83b2-acaefc0bd727-hosts-file\") pod \"node-resolver-lpvwr\" (UID: \"fd7db434-46fc-4a27-83b2-acaefc0bd727\") " pod="openshift-dns/node-resolver-lpvwr" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353202 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7db434-46fc-4a27-83b2-acaefc0bd727-tmp-dir\") pod \"node-resolver-lpvwr\" (UID: \"fd7db434-46fc-4a27-83b2-acaefc0bd727\") " pod="openshift-dns/node-resolver-lpvwr" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353216 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs\") pod \"network-metrics-daemon-bh66q\" (UID: \"a2c6f5f5-7f01-421d-bb94-170b7a1cda67\") " pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353229 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-run-netns\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353267 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-registration-dir\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353302 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmhw2\" (UniqueName: \"kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2\") pod \"network-check-target-xppkv\" (UID: \"b0396d27-4305-4393-8d0e-f9a5cff12337\") " pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353332 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-multus-cni-dir\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353351 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-etc-kubernetes\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.353394 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353366 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15d09c81-aa45-4b25-83c2-c13f788e021c-env-overrides\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353389 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-sysconfig\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353412 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-sys\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353434 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353449 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-var-lib-cni-multus\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353471 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rpq\" (UniqueName: \"kubernetes.io/projected/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-kube-api-access-j8rpq\") pod \"network-metrics-daemon-bh66q\" (UID: \"a2c6f5f5-7f01-421d-bb94-170b7a1cda67\") " pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353489 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/839de40a-5fee-44af-8173-a65990f845cf-host\") pod \"node-ca-knrrs\" (UID: \"839de40a-5fee-44af-8173-a65990f845cf\") " pod="openshift-image-registry/node-ca-knrrs" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353511 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-sysctl-d\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353537 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c736c89e-7b1b-4d96-b36e-60956aca58af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353560 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-device-dir\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353575 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-run-multus-certs\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353597 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-kubelet\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353615 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-run-ovn-kubernetes\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353629 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-sysctl-conf\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353666 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-sys-fs\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353694 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgdp9\" (UniqueName: \"kubernetes.io/projected/548df819-809f-4a00-bc11-2758f97617ed-kube-api-access-pgdp9\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.353975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353720 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-multus-socket-dir-parent\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353744 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-slash\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353767 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-run-ovn\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353789 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15d09c81-aa45-4b25-83c2-c13f788e021c-ovnkube-config\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353817 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c736c89e-7b1b-4d96-b36e-60956aca58af-cnibin\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353834 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-node-log\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353853 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-cni-netd\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353868 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrv7r\" (UniqueName: \"kubernetes.io/projected/cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378-kube-api-access-wrv7r\") pod \"iptables-alerter-ck7z5\" (UID: \"cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378\") " pod="openshift-network-operator/iptables-alerter-ck7z5" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353899 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-var-lib-kubelet\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353925 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-run-systemd\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353940 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z22d6\" (UniqueName: \"kubernetes.io/projected/15d09c81-aa45-4b25-83c2-c13f788e021c-kube-api-access-z22d6\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353957 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-run\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353973 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c736c89e-7b1b-4d96-b36e-60956aca58af-cni-binary-copy\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.353991 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-log-socket\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.354006 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-hostroot\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.354018 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-systemd-units\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.354541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.354032 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-run-openvswitch\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.355014 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.354046 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4bv8\" (UniqueName: \"kubernetes.io/projected/87dcc8db-8293-45b1-bb67-1db76191f899-kube-api-access-k4bv8\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.355014 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.354080 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49d42\" (UniqueName: \"kubernetes.io/projected/839de40a-5fee-44af-8173-a65990f845cf-kube-api-access-49d42\") pod \"node-ca-knrrs\" (UID: \"839de40a-5fee-44af-8173-a65990f845cf\") " pod="openshift-image-registry/node-ca-knrrs" Jul 14 19:46:23.355014 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.354095 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c736c89e-7b1b-4d96-b36e-60956aca58af-os-release\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.355014 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.354122 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-system-cni-dir\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.355014 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.354135 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-var-lib-kubelet\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.355014 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.354155 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378-iptables-alerter-script\") pod \"iptables-alerter-ck7z5\" (UID: \"cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378\") " pod="openshift-network-operator/iptables-alerter-ck7z5" Jul 14 19:46:23.355014 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.354169 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/322d4ffe-4783-4a5a-a18f-47878e117000-konnectivity-ca\") pod \"konnectivity-agent-ff7s5\" (UID: \"322d4ffe-4783-4a5a-a18f-47878e117000\") " pod="kube-system/konnectivity-agent-ff7s5" Jul 14 19:46:23.355014 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.354184 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3f090a3-021a-4185-bcc8-7c00d7d56b77-dbus\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:23.355014 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.354208 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-cnibin\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.355014 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.354237 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.355014 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.354264 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c736c89e-7b1b-4d96-b36e-60956aca58af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.355014 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.354288 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c736c89e-7b1b-4d96-b36e-60956aca58af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.358708 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.358670 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal" event={"ID":"4baf446df5fec806f69a038fd9e73461","Type":"ContainerStarted","Data":"a7505e2c8bfec73ab7a91c0d739c3a5ef0f3c5d9e1415ef17b9eb5d68f047ef2"} Jul 14 19:46:23.359665 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.359645 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.ec2.internal" event={"ID":"130523dbb878b2a1af0eb57e85edc42c","Type":"ContainerStarted","Data":"406a88a6269e0b2846b9f4930dd34140fe224b65284b2b37a84fde98e6289a79"} Jul 14 19:46:23.454804 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.454774 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.454804 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.454801 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.455042 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.454817 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd7db434-46fc-4a27-83b2-acaefc0bd727-hosts-file\") pod \"node-resolver-lpvwr\" (UID: \"fd7db434-46fc-4a27-83b2-acaefc0bd727\") " pod="openshift-dns/node-resolver-lpvwr" Jul 14 19:46:23.455042 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.454832 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7db434-46fc-4a27-83b2-acaefc0bd727-tmp-dir\") pod \"node-resolver-lpvwr\" (UID: \"fd7db434-46fc-4a27-83b2-acaefc0bd727\") " pod="openshift-dns/node-resolver-lpvwr" Jul 14 19:46:23.455042 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.454900 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd7db434-46fc-4a27-83b2-acaefc0bd727-hosts-file\") pod \"node-resolver-lpvwr\" (UID: \"fd7db434-46fc-4a27-83b2-acaefc0bd727\") " pod="openshift-dns/node-resolver-lpvwr" Jul 14 19:46:23.455042 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.454915 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.455042 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.454916 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs\") pod \"network-metrics-daemon-bh66q\" (UID: \"a2c6f5f5-7f01-421d-bb94-170b7a1cda67\") " pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:23.455042 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.454936 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.455042 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.454987 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-run-netns\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.455042 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455005 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-registration-dir\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.455042 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455021 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhw2\" (UniqueName: \"kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2\") pod \"network-check-target-xppkv\" (UID: \"b0396d27-4305-4393-8d0e-f9a5cff12337\") " pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:23.455042 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.455033 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455098 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-multus-cni-dir\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455037 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-multus-cni-dir\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455113 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-registration-dir\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455130 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-run-netns\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455150 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7db434-46fc-4a27-83b2-acaefc0bd727-tmp-dir\") pod \"node-resolver-lpvwr\" (UID: \"fd7db434-46fc-4a27-83b2-acaefc0bd727\") " pod="openshift-dns/node-resolver-lpvwr" Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.455164 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs podName:a2c6f5f5-7f01-421d-bb94-170b7a1cda67 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:23.955100503 +0000 UTC m=+2.034846288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs") pod "network-metrics-daemon-bh66q" (UID: "a2c6f5f5-7f01-421d-bb94-170b7a1cda67") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455219 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-etc-kubernetes\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455241 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15d09c81-aa45-4b25-83c2-c13f788e021c-env-overrides\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455255 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-sysconfig\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455272 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-sys\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455292 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-etc-kubernetes\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455302 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455327 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-sys\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455330 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-var-lib-cni-multus\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455327 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-sysconfig\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.455393 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455386 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rpq\" (UniqueName: \"kubernetes.io/projected/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-kube-api-access-j8rpq\") pod \"network-metrics-daemon-bh66q\" (UID: \"a2c6f5f5-7f01-421d-bb94-170b7a1cda67\") " pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:23.455517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455426 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/839de40a-5fee-44af-8173-a65990f845cf-host\") pod \"node-ca-knrrs\" (UID: \"839de40a-5fee-44af-8173-a65990f845cf\") " pod="openshift-image-registry/node-ca-knrrs" Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.455465 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret podName:d3f090a3-021a-4185-bcc8-7c00d7d56b77 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:23.955452731 +0000 UTC m=+2.035198505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret") pod "global-pull-secret-syncer-6w8gx" (UID: "d3f090a3-021a-4185-bcc8-7c00d7d56b77") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455485 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/839de40a-5fee-44af-8173-a65990f845cf-host\") pod \"node-ca-knrrs\" (UID: \"839de40a-5fee-44af-8173-a65990f845cf\") " pod="openshift-image-registry/node-ca-knrrs" Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455486 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-sysctl-d\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455485 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-var-lib-cni-multus\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455578 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c736c89e-7b1b-4d96-b36e-60956aca58af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455610 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-device-dir\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455643 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-run-multus-certs\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455674 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-kubelet\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455581 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-sysctl-d\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455700 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-device-dir\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455701 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-run-ovn-kubernetes\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455735 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-run-ovn-kubernetes\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455743 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-sysctl-conf\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455710 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-run-multus-certs\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455750 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/15d09c81-aa45-4b25-83c2-c13f788e021c-env-overrides\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.456182 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455758 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-kubelet\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455778 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-sys-fs\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455750 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c736c89e-7b1b-4d96-b36e-60956aca58af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455829 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-sys-fs\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455857 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgdp9\" (UniqueName: \"kubernetes.io/projected/548df819-809f-4a00-bc11-2758f97617ed-kube-api-access-pgdp9\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455872 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-sysctl-conf\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455882 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-multus-socket-dir-parent\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455915 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-slash\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456109 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-multus-socket-dir-parent\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.455974 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-slash\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456278 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-run-ovn\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456322 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15d09c81-aa45-4b25-83c2-c13f788e021c-ovnkube-config\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456380 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c736c89e-7b1b-4d96-b36e-60956aca58af-cnibin\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456401 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-run-ovn\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456420 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-node-log\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456471 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-node-log\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456482 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-cni-netd\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.456864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456528 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c736c89e-7b1b-4d96-b36e-60956aca58af-cnibin\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456569 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrv7r\" (UniqueName: \"kubernetes.io/projected/cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378-kube-api-access-wrv7r\") pod \"iptables-alerter-ck7z5\" (UID: \"cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378\") " pod="openshift-network-operator/iptables-alerter-ck7z5" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456607 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-var-lib-kubelet\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456634 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-cni-netd\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456639 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-run-systemd\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456687 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-run-systemd\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456687 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z22d6\" (UniqueName: \"kubernetes.io/projected/15d09c81-aa45-4b25-83c2-c13f788e021c-kube-api-access-z22d6\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456743 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-run\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456778 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c736c89e-7b1b-4d96-b36e-60956aca58af-cni-binary-copy\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456829 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-log-socket\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456858 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-hostroot\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456887 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-systemd-units\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456916 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-run-openvswitch\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456950 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4bv8\" (UniqueName: \"kubernetes.io/projected/87dcc8db-8293-45b1-bb67-1db76191f899-kube-api-access-k4bv8\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456981 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49d42\" (UniqueName: \"kubernetes.io/projected/839de40a-5fee-44af-8173-a65990f845cf-kube-api-access-49d42\") pod \"node-ca-knrrs\" (UID: \"839de40a-5fee-44af-8173-a65990f845cf\") " pod="openshift-image-registry/node-ca-knrrs" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457017 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c736c89e-7b1b-4d96-b36e-60956aca58af-os-release\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457081 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-system-cni-dir\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.457690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457139 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-var-lib-kubelet\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457174 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378-iptables-alerter-script\") pod \"iptables-alerter-ck7z5\" (UID: \"cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378\") " pod="openshift-network-operator/iptables-alerter-ck7z5" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457165 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/15d09c81-aa45-4b25-83c2-c13f788e021c-ovnkube-config\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457257 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-systemd-units\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457254 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/322d4ffe-4783-4a5a-a18f-47878e117000-konnectivity-ca\") pod \"konnectivity-agent-ff7s5\" (UID: \"322d4ffe-4783-4a5a-a18f-47878e117000\") " pod="kube-system/konnectivity-agent-ff7s5" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457325 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-run\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457316 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-run-openvswitch\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.456742 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-var-lib-kubelet\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457436 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3f090a3-021a-4185-bcc8-7c00d7d56b77-dbus\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457367 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d3f090a3-021a-4185-bcc8-7c00d7d56b77-dbus\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457893 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c736c89e-7b1b-4d96-b36e-60956aca58af-cni-binary-copy\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457896 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/322d4ffe-4783-4a5a-a18f-47878e117000-konnectivity-ca\") pod \"konnectivity-agent-ff7s5\" (UID: \"322d4ffe-4783-4a5a-a18f-47878e117000\") " pod="kube-system/konnectivity-agent-ff7s5" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457961 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-cnibin\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457979 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-log-socket\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458014 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-var-lib-kubelet\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458012 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458092 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c736c89e-7b1b-4d96-b36e-60956aca58af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.458460 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458115 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c736c89e-7b1b-4d96-b36e-60956aca58af-os-release\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458134 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c736c89e-7b1b-4d96-b36e-60956aca58af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458174 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458182 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-os-release\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458232 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-cnibin\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458253 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-os-release\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458271 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-var-lib-cni-bin\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458045 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-hostroot\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458307 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15d09c81-aa45-4b25-83c2-c13f788e021c-ovnkube-script-lib\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458354 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-host\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458385 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/87dcc8db-8293-45b1-bb67-1db76191f899-etc-tuned\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458426 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/839de40a-5fee-44af-8173-a65990f845cf-serviceca\") pod \"node-ca-knrrs\" (UID: \"839de40a-5fee-44af-8173-a65990f845cf\") " pod="openshift-image-registry/node-ca-knrrs" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458462 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-etc-openvswitch\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458516 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-lib-modules\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458543 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87dcc8db-8293-45b1-bb67-1db76191f899-tmp\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458588 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3f090a3-021a-4185-bcc8-7c00d7d56b77-kubelet-config\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458610 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-cni-binary-copy\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458624 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378-iptables-alerter-script\") pod \"iptables-alerter-ck7z5\" (UID: \"cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378\") " pod="openshift-network-operator/iptables-alerter-ck7z5" Jul 14 19:46:23.459210 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458682 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-cni-bin\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458743 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c736c89e-7b1b-4d96-b36e-60956aca58af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458769 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-modprobe-d\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458805 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-var-lib-cni-bin\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458833 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-socket-dir\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458880 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-kubernetes\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458928 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-modprobe-d\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458909 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c736c89e-7b1b-4d96-b36e-60956aca58af-system-cni-dir\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458968 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c736c89e-7b1b-4d96-b36e-60956aca58af-system-cni-dir\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.458975 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdkvq\" (UniqueName: \"kubernetes.io/projected/c736c89e-7b1b-4d96-b36e-60956aca58af-kube-api-access-tdkvq\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.457961 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-system-cni-dir\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459009 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pddkq\" (UniqueName: \"kubernetes.io/projected/fd7db434-46fc-4a27-83b2-acaefc0bd727-kube-api-access-pddkq\") pod \"node-resolver-lpvwr\" (UID: \"fd7db434-46fc-4a27-83b2-acaefc0bd727\") " pod="openshift-dns/node-resolver-lpvwr" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459037 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-run-k8s-cni-cncf-io\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459085 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-run-netns\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459117 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-multus-conf-dir\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459108 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/548df819-809f-4a00-bc11-2758f97617ed-socket-dir\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459154 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-multus-daemon-config\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.459998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459189 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h7cp\" (UniqueName: \"kubernetes.io/projected/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-kube-api-access-5h7cp\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459187 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-kubernetes\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459218 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-var-lib-openvswitch\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459250 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15d09c81-aa45-4b25-83c2-c13f788e021c-ovn-node-metrics-cert\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459272 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-cni-binary-copy\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459284 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378-host-slash\") pod \"iptables-alerter-ck7z5\" (UID: \"cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378\") " pod="openshift-network-operator/iptables-alerter-ck7z5" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459314 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/322d4ffe-4783-4a5a-a18f-47878e117000-agent-certs\") pod \"konnectivity-agent-ff7s5\" (UID: \"322d4ffe-4783-4a5a-a18f-47878e117000\") " pod="kube-system/konnectivity-agent-ff7s5" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459338 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-host\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459348 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-systemd\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459390 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-etc-openvswitch\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459411 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-lib-modules\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459452 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-host-cni-bin\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459524 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/87dcc8db-8293-45b1-bb67-1db76191f899-etc-systemd\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459708 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-run-k8s-cni-cncf-io\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459867 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-host-run-netns\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459937 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-multus-conf-dir\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.459935 2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.460231 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378-host-slash\") pod \"iptables-alerter-ck7z5\" (UID: \"cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378\") " pod="openshift-network-operator/iptables-alerter-ck7z5" Jul 14 19:46:23.460779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.460367 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/839de40a-5fee-44af-8173-a65990f845cf-serviceca\") pod \"node-ca-knrrs\" (UID: \"839de40a-5fee-44af-8173-a65990f845cf\") " pod="openshift-image-registry/node-ca-knrrs" Jul 14 19:46:23.461607 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.460367 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d3f090a3-021a-4185-bcc8-7c00d7d56b77-kubelet-config\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:23.461607 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.460804 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/15d09c81-aa45-4b25-83c2-c13f788e021c-ovnkube-script-lib\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.461607 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.460807 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c736c89e-7b1b-4d96-b36e-60956aca58af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.461607 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.460933 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/15d09c81-aa45-4b25-83c2-c13f788e021c-var-lib-openvswitch\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.461797 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.461615 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-multus-daemon-config\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.462539 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.462518 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:23.462539 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.462544 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:23.462539 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.462558 2557 projected.go:196] Error preparing data for projected volume kube-api-access-qmhw2 for pod openshift-network-diagnostics/network-check-target-xppkv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:23.462786 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.462680 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2 podName:b0396d27-4305-4393-8d0e-f9a5cff12337 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:23.962662558 +0000 UTC m=+2.042408345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qmhw2" (UniqueName: "kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2") pod "network-check-target-xppkv" (UID: "b0396d27-4305-4393-8d0e-f9a5cff12337") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:23.463672 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.463647 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87dcc8db-8293-45b1-bb67-1db76191f899-tmp\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.463772 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.463677 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/87dcc8db-8293-45b1-bb67-1db76191f899-etc-tuned\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.463772 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.463723 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/15d09c81-aa45-4b25-83c2-c13f788e021c-ovn-node-metrics-cert\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.463899 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.463884 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/322d4ffe-4783-4a5a-a18f-47878e117000-agent-certs\") pod \"konnectivity-agent-ff7s5\" (UID: \"322d4ffe-4783-4a5a-a18f-47878e117000\") " pod="kube-system/konnectivity-agent-ff7s5" Jul 14 19:46:23.464441 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.464425 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rpq\" (UniqueName: \"kubernetes.io/projected/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-kube-api-access-j8rpq\") pod \"network-metrics-daemon-bh66q\" (UID: \"a2c6f5f5-7f01-421d-bb94-170b7a1cda67\") " pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:23.466520 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.466492 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z22d6\" (UniqueName: \"kubernetes.io/projected/15d09c81-aa45-4b25-83c2-c13f788e021c-kube-api-access-z22d6\") pod \"ovnkube-node-926lj\" (UID: \"15d09c81-aa45-4b25-83c2-c13f788e021c\") " pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.466975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.466955 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrv7r\" (UniqueName: \"kubernetes.io/projected/cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378-kube-api-access-wrv7r\") pod \"iptables-alerter-ck7z5\" (UID: \"cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378\") " pod="openshift-network-operator/iptables-alerter-ck7z5" Jul 14 19:46:23.467344 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.467325 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49d42\" (UniqueName: \"kubernetes.io/projected/839de40a-5fee-44af-8173-a65990f845cf-kube-api-access-49d42\") pod \"node-ca-knrrs\" (UID: \"839de40a-5fee-44af-8173-a65990f845cf\") " pod="openshift-image-registry/node-ca-knrrs" Jul 14 19:46:23.467417 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.467342 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgdp9\" (UniqueName: \"kubernetes.io/projected/548df819-809f-4a00-bc11-2758f97617ed-kube-api-access-pgdp9\") pod \"aws-ebs-csi-driver-node-v2w8r\" (UID: \"548df819-809f-4a00-bc11-2758f97617ed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.467675 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.467655 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4bv8\" (UniqueName: \"kubernetes.io/projected/87dcc8db-8293-45b1-bb67-1db76191f899-kube-api-access-k4bv8\") pod \"tuned-d7bzh\" (UID: \"87dcc8db-8293-45b1-bb67-1db76191f899\") " pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.476487 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.476469 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pddkq\" (UniqueName: \"kubernetes.io/projected/fd7db434-46fc-4a27-83b2-acaefc0bd727-kube-api-access-pddkq\") pod \"node-resolver-lpvwr\" (UID: \"fd7db434-46fc-4a27-83b2-acaefc0bd727\") " pod="openshift-dns/node-resolver-lpvwr" Jul 14 19:46:23.476862 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.476843 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdkvq\" (UniqueName: \"kubernetes.io/projected/c736c89e-7b1b-4d96-b36e-60956aca58af-kube-api-access-tdkvq\") pod \"multus-additional-cni-plugins-pdjn9\" (UID: \"c736c89e-7b1b-4d96-b36e-60956aca58af\") " pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.481565 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.481547 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h7cp\" (UniqueName: \"kubernetes.io/projected/0e58d2f7-1d92-4bb0-aadd-83b5ecb40331-kube-api-access-5h7cp\") pod \"multus-rqfrv\" (UID: \"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331\") " pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.492931 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.492912 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:23.571868 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.571794 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rqfrv" Jul 14 19:46:23.579887 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:23.579863 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e58d2f7_1d92_4bb0_aadd_83b5ecb40331.slice/crio-2f8f3fbf46ae50418b222a4db907509c1c90090038e7c51fa9d9faaa51a68796 WatchSource:0}: Error finding container 2f8f3fbf46ae50418b222a4db907509c1c90090038e7c51fa9d9faaa51a68796: Status 404 returned error can't find the container with id 2f8f3fbf46ae50418b222a4db907509c1c90090038e7c51fa9d9faaa51a68796 Jul 14 19:46:23.612258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.612231 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ck7z5" Jul 14 19:46:23.618822 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.618805 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ff7s5" Jul 14 19:46:23.619096 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:23.619077 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfcbd9c8_4dbd_4ebc_b0be_0ee1f2e70378.slice/crio-eeeb81cdb314a9ca930741795cfacc86668f3afb681cf16874be68d1403bffd4 WatchSource:0}: Error finding container eeeb81cdb314a9ca930741795cfacc86668f3afb681cf16874be68d1403bffd4: Status 404 returned error can't find the container with id eeeb81cdb314a9ca930741795cfacc86668f3afb681cf16874be68d1403bffd4 Jul 14 19:46:23.623516 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.623488 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" Jul 14 19:46:23.628528 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:23.628502 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322d4ffe_4783_4a5a_a18f_47878e117000.slice/crio-18ce31d296ffbd2c8afac5276303208239d3f40c694b83865a16d2c57355defe WatchSource:0}: Error finding container 18ce31d296ffbd2c8afac5276303208239d3f40c694b83865a16d2c57355defe: Status 404 returned error can't find the container with id 18ce31d296ffbd2c8afac5276303208239d3f40c694b83865a16d2c57355defe Jul 14 19:46:23.629367 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.629344 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lpvwr" Jul 14 19:46:23.631251 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:23.631226 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548df819_809f_4a00_bc11_2758f97617ed.slice/crio-c2c87fa177ef8c288c2d2755af496414af040c57ee4f11bc62ee328ff2f6016a WatchSource:0}: Error finding container c2c87fa177ef8c288c2d2755af496414af040c57ee4f11bc62ee328ff2f6016a: Status 404 returned error can't find the container with id c2c87fa177ef8c288c2d2755af496414af040c57ee4f11bc62ee328ff2f6016a Jul 14 19:46:23.634901 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.634876 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:23.636197 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:23.636170 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7db434_46fc_4a27_83b2_acaefc0bd727.slice/crio-f6207a6928c03a041cd9c2f3790c49aa8f0044af9e556a9d9902dddd15f96e57 WatchSource:0}: Error finding container f6207a6928c03a041cd9c2f3790c49aa8f0044af9e556a9d9902dddd15f96e57: Status 404 returned error can't find the container with id f6207a6928c03a041cd9c2f3790c49aa8f0044af9e556a9d9902dddd15f96e57 Jul 14 19:46:23.640701 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.640683 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" Jul 14 19:46:23.642555 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:23.642535 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d09c81_aa45_4b25_83c2_c13f788e021c.slice/crio-8ddff1430b32704741a51bf374da4695c4536c9b8dd4dd8248dd05a5d7d399a0 WatchSource:0}: Error finding container 8ddff1430b32704741a51bf374da4695c4536c9b8dd4dd8248dd05a5d7d399a0: Status 404 returned error can't find the container with id 8ddff1430b32704741a51bf374da4695c4536c9b8dd4dd8248dd05a5d7d399a0 Jul 14 19:46:23.645312 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.645294 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-knrrs" Jul 14 19:46:23.648446 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:23.648425 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87dcc8db_8293_45b1_bb67_1db76191f899.slice/crio-9954a37bddda2b3244ba0a5be22bd9efed51ad3bcc89dc0faa6d132ef532867d WatchSource:0}: Error finding container 9954a37bddda2b3244ba0a5be22bd9efed51ad3bcc89dc0faa6d132ef532867d: Status 404 returned error can't find the container with id 9954a37bddda2b3244ba0a5be22bd9efed51ad3bcc89dc0faa6d132ef532867d Jul 14 19:46:23.649581 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.649561 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pdjn9" Jul 14 19:46:23.653359 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:23.653342 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839de40a_5fee_44af_8173_a65990f845cf.slice/crio-84f4265e58fd1d18db64a0adadcc72a6a0055add36fa599432a5d8810ccbb2bb WatchSource:0}: Error finding container 84f4265e58fd1d18db64a0adadcc72a6a0055add36fa599432a5d8810ccbb2bb: Status 404 returned error can't find the container with id 84f4265e58fd1d18db64a0adadcc72a6a0055add36fa599432a5d8810ccbb2bb Jul 14 19:46:23.657919 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:46:23.657901 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc736c89e_7b1b_4d96_b36e_60956aca58af.slice/crio-60c8d111a55c102387da3801dddb39f33e9c20481dcf212260d6b120ef450bc1 WatchSource:0}: Error finding container 60c8d111a55c102387da3801dddb39f33e9c20481dcf212260d6b120ef450bc1: Status 404 returned error can't find the container with id 60c8d111a55c102387da3801dddb39f33e9c20481dcf212260d6b120ef450bc1 Jul 14 19:46:23.962465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.962387 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs\") pod \"network-metrics-daemon-bh66q\" (UID: \"a2c6f5f5-7f01-421d-bb94-170b7a1cda67\") " pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:23.962465 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:23.962449 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:23.962678 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.962605 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:23.962678 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.962608 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:23.962678 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.962664 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret podName:d3f090a3-021a-4185-bcc8-7c00d7d56b77 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:24.962646648 +0000 UTC m=+3.042392432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret") pod "global-pull-secret-syncer-6w8gx" (UID: "d3f090a3-021a-4185-bcc8-7c00d7d56b77") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:23.962827 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:23.962684 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs podName:a2c6f5f5-7f01-421d-bb94-170b7a1cda67 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:24.962674377 +0000 UTC m=+3.042420158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs") pod "network-metrics-daemon-bh66q" (UID: "a2c6f5f5-7f01-421d-bb94-170b7a1cda67") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:24.063825 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.063790 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhw2\" (UniqueName: \"kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2\") pod \"network-check-target-xppkv\" (UID: \"b0396d27-4305-4393-8d0e-f9a5cff12337\") " pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:24.064019 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:24.064000 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:24.064196 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:24.064024 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:24.064196 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:24.064037 2557 projected.go:196] Error preparing data for projected volume kube-api-access-qmhw2 for pod openshift-network-diagnostics/network-check-target-xppkv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:24.064196 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:24.064112 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2 podName:b0396d27-4305-4393-8d0e-f9a5cff12337 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:25.064092925 +0000 UTC m=+3.143838740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmhw2" (UniqueName: "kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2") pod "network-check-target-xppkv" (UID: "b0396d27-4305-4393-8d0e-f9a5cff12337") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:24.307446 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.307354 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 19:41:23 +0000 UTC" deadline="2028-02-23 12:05:59.237892087 +0000 UTC" Jul 14 19:46:24.307446 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.307395 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14128h19m34.930501571s" Jul 14 19:46:24.357390 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.357356 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:24.357574 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:24.357509 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:24.366692 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.366655 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pdjn9" event={"ID":"c736c89e-7b1b-4d96-b36e-60956aca58af","Type":"ContainerStarted","Data":"60c8d111a55c102387da3801dddb39f33e9c20481dcf212260d6b120ef450bc1"} Jul 14 19:46:24.372528 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.372450 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" event={"ID":"87dcc8db-8293-45b1-bb67-1db76191f899","Type":"ContainerStarted","Data":"9954a37bddda2b3244ba0a5be22bd9efed51ad3bcc89dc0faa6d132ef532867d"} Jul 14 19:46:24.375235 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.375125 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" event={"ID":"15d09c81-aa45-4b25-83c2-c13f788e021c","Type":"ContainerStarted","Data":"8ddff1430b32704741a51bf374da4695c4536c9b8dd4dd8248dd05a5d7d399a0"} Jul 14 19:46:24.376711 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.376612 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ck7z5" event={"ID":"cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378","Type":"ContainerStarted","Data":"eeeb81cdb314a9ca930741795cfacc86668f3afb681cf16874be68d1403bffd4"} Jul 14 19:46:24.380206 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.380124 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rqfrv" event={"ID":"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331","Type":"ContainerStarted","Data":"2f8f3fbf46ae50418b222a4db907509c1c90090038e7c51fa9d9faaa51a68796"} Jul 14 19:46:24.384362 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.384338 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-knrrs" event={"ID":"839de40a-5fee-44af-8173-a65990f845cf","Type":"ContainerStarted","Data":"84f4265e58fd1d18db64a0adadcc72a6a0055add36fa599432a5d8810ccbb2bb"} Jul 14 19:46:24.386840 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.386816 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lpvwr" event={"ID":"fd7db434-46fc-4a27-83b2-acaefc0bd727","Type":"ContainerStarted","Data":"f6207a6928c03a041cd9c2f3790c49aa8f0044af9e556a9d9902dddd15f96e57"} Jul 14 19:46:24.394115 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.394088 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" event={"ID":"548df819-809f-4a00-bc11-2758f97617ed","Type":"ContainerStarted","Data":"c2c87fa177ef8c288c2d2755af496414af040c57ee4f11bc62ee328ff2f6016a"} Jul 14 19:46:24.401416 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.398417 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ff7s5" event={"ID":"322d4ffe-4783-4a5a-a18f-47878e117000","Type":"ContainerStarted","Data":"18ce31d296ffbd2c8afac5276303208239d3f40c694b83865a16d2c57355defe"} Jul 14 19:46:24.712698 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.712617 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:24.969342 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.969253 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:24.969552 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:24.969364 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs\") pod \"network-metrics-daemon-bh66q\" (UID: \"a2c6f5f5-7f01-421d-bb94-170b7a1cda67\") " pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:24.969552 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:24.969508 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:24.969660 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:24.969576 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs podName:a2c6f5f5-7f01-421d-bb94-170b7a1cda67 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:26.969552029 +0000 UTC m=+5.049297822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs") pod "network-metrics-daemon-bh66q" (UID: "a2c6f5f5-7f01-421d-bb94-170b7a1cda67") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:24.969660 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:24.969641 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:24.969773 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:24.969675 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret podName:d3f090a3-021a-4185-bcc8-7c00d7d56b77 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:26.969662156 +0000 UTC m=+5.049407933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret") pod "global-pull-secret-syncer-6w8gx" (UID: "d3f090a3-021a-4185-bcc8-7c00d7d56b77") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:25.070859 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:25.070821 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhw2\" (UniqueName: \"kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2\") pod \"network-check-target-xppkv\" (UID: \"b0396d27-4305-4393-8d0e-f9a5cff12337\") " pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:25.075413 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:25.075382 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:25.075565 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:25.075421 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:25.075565 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:25.075457 2557 projected.go:196] Error preparing data for projected volume kube-api-access-qmhw2 for pod openshift-network-diagnostics/network-check-target-xppkv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:25.075664 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:25.075568 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2 podName:b0396d27-4305-4393-8d0e-f9a5cff12337 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:27.075538618 +0000 UTC m=+5.155284395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmhw2" (UniqueName: "kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2") pod "network-check-target-xppkv" (UID: "b0396d27-4305-4393-8d0e-f9a5cff12337") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:25.206767 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:25.206678 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:25.308611 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:25.308496 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 19:41:23 +0000 UTC" deadline="2028-01-16 00:08:24.64276672 +0000 UTC" Jul 14 19:46:25.308611 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:25.308562 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13204h21m59.334208924s" Jul 14 19:46:25.354865 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:25.354830 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:25.355052 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:25.354963 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:25.355451 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:25.355414 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:25.355565 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:25.355518 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:26.354785 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:26.354716 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:26.355232 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:26.354859 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:26.987141 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:26.986438 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs\") pod \"network-metrics-daemon-bh66q\" (UID: \"a2c6f5f5-7f01-421d-bb94-170b7a1cda67\") " pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:26.987141 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:26.986498 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:26.987141 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:26.986631 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:26.987141 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:26.986703 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret podName:d3f090a3-021a-4185-bcc8-7c00d7d56b77 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:30.986674024 +0000 UTC m=+9.066419796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret") pod "global-pull-secret-syncer-6w8gx" (UID: "d3f090a3-021a-4185-bcc8-7c00d7d56b77") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:26.987141 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:26.987143 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:26.987525 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:26.987209 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs podName:a2c6f5f5-7f01-421d-bb94-170b7a1cda67 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:30.987191703 +0000 UTC m=+9.066937476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs") pod "network-metrics-daemon-bh66q" (UID: "a2c6f5f5-7f01-421d-bb94-170b7a1cda67") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:27.087483 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:27.087349 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhw2\" (UniqueName: \"kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2\") pod \"network-check-target-xppkv\" (UID: \"b0396d27-4305-4393-8d0e-f9a5cff12337\") " pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:27.087673 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:27.087553 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:27.087673 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:27.087586 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:27.087673 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:27.087599 2557 projected.go:196] Error preparing data for projected volume kube-api-access-qmhw2 for pod openshift-network-diagnostics/network-check-target-xppkv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:27.087673 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:27.087659 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2 podName:b0396d27-4305-4393-8d0e-f9a5cff12337 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:31.087640404 +0000 UTC m=+9.167386196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmhw2" (UniqueName: "kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2") pod "network-check-target-xppkv" (UID: "b0396d27-4305-4393-8d0e-f9a5cff12337") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:27.355308 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:27.354583 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:27.355308 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:27.354705 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:27.355308 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:27.355103 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:27.355308 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:27.355210 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:28.355582 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:28.355512 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:28.356171 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:28.355654 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:29.355162 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:29.355051 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:29.355162 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:29.355082 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:29.355370 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:29.355204 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:29.355370 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:29.355270 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:30.358069 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:30.357548 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:30.358069 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:30.357694 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:31.022389 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:31.022343 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs\") pod \"network-metrics-daemon-bh66q\" (UID: \"a2c6f5f5-7f01-421d-bb94-170b7a1cda67\") " pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:31.022577 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:31.022408 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:31.022577 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:31.022510 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:31.022577 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:31.022525 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:31.022735 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:31.022580 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret podName:d3f090a3-021a-4185-bcc8-7c00d7d56b77 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:39.022562622 +0000 UTC m=+17.102308398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret") pod "global-pull-secret-syncer-6w8gx" (UID: "d3f090a3-021a-4185-bcc8-7c00d7d56b77") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:31.022735 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:31.022602 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs podName:a2c6f5f5-7f01-421d-bb94-170b7a1cda67 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:39.022590716 +0000 UTC m=+17.102336493 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs") pod "network-metrics-daemon-bh66q" (UID: "a2c6f5f5-7f01-421d-bb94-170b7a1cda67") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:31.123363 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:31.122937 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhw2\" (UniqueName: \"kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2\") pod \"network-check-target-xppkv\" (UID: \"b0396d27-4305-4393-8d0e-f9a5cff12337\") " pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:31.123363 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:31.123142 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:31.123363 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:31.123162 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:31.123363 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:31.123174 2557 projected.go:196] Error preparing data for projected volume kube-api-access-qmhw2 for pod openshift-network-diagnostics/network-check-target-xppkv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:31.123363 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:31.123322 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2 podName:b0396d27-4305-4393-8d0e-f9a5cff12337 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:39.123302392 +0000 UTC m=+17.203048163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmhw2" (UniqueName: "kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2") pod "network-check-target-xppkv" (UID: "b0396d27-4305-4393-8d0e-f9a5cff12337") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:31.355675 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:31.355258 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:31.355675 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:31.355384 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:31.355675 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:31.355498 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:31.355675 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:31.355582 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:32.356203 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:32.356168 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:32.356634 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:32.356291 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:33.355667 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:33.355447 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:33.355856 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:33.355451 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:33.355856 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:33.355757 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:33.355963 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:33.355855 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:34.354799 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:34.354760 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:34.355278 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:34.354909 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:35.355329 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:35.355286 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:35.355749 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:35.355430 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:35.355749 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:35.355491 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:35.355749 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:35.355621 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:36.358010 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:36.357977 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:36.358453 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:36.358129 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:37.355001 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:37.354962 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:37.355199 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:37.354962 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:37.355199 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:37.355107 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:37.355312 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:37.355208 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:38.355030 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:38.355000 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:38.355494 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:38.355146 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:39.081291 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:39.081258 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs\") pod \"network-metrics-daemon-bh66q\" (UID: \"a2c6f5f5-7f01-421d-bb94-170b7a1cda67\") " pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:39.081518 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:39.081313 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:39.081518 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:39.081425 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:39.081518 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:39.081487 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:39.081518 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:39.081517 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs podName:a2c6f5f5-7f01-421d-bb94-170b7a1cda67 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:55.081494267 +0000 UTC m=+33.161240061 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs") pod "network-metrics-daemon-bh66q" (UID: "a2c6f5f5-7f01-421d-bb94-170b7a1cda67") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:39.081718 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:39.081542 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret podName:d3f090a3-021a-4185-bcc8-7c00d7d56b77 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:55.081525861 +0000 UTC m=+33.161271632 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret") pod "global-pull-secret-syncer-6w8gx" (UID: "d3f090a3-021a-4185-bcc8-7c00d7d56b77") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:39.182315 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:39.182279 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhw2\" (UniqueName: \"kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2\") pod \"network-check-target-xppkv\" (UID: \"b0396d27-4305-4393-8d0e-f9a5cff12337\") " pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:39.182497 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:39.182433 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:39.182497 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:39.182449 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:39.182497 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:39.182458 2557 projected.go:196] Error preparing data for projected volume kube-api-access-qmhw2 for pod openshift-network-diagnostics/network-check-target-xppkv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:39.182657 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:39.182512 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2 podName:b0396d27-4305-4393-8d0e-f9a5cff12337 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:55.182495911 +0000 UTC m=+33.262241684 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmhw2" (UniqueName: "kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2") pod "network-check-target-xppkv" (UID: "b0396d27-4305-4393-8d0e-f9a5cff12337") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:39.354696 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:39.354619 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:39.354696 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:39.354648 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:39.354898 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:39.354734 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:39.354898 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:39.354814 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:40.355322 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:40.355288 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:40.355716 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:40.355403 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:41.354878 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:41.354834 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:41.355083 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:41.354834 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:41.355083 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:41.354975 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:41.355083 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:41.355034 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:42.357096 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:42.357071 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:42.357446 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:42.357188 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:43.355147 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.354809 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:43.355286 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.354846 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:43.355286 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:43.355247 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:43.355286 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:43.355279 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:43.440220 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.440140 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.ec2.internal" event={"ID":"130523dbb878b2a1af0eb57e85edc42c","Type":"ContainerStarted","Data":"f7a48d28681f523835ef7e921c588cd526f46aa1c8c3c5abf85d8093dc6cdf2a"} Jul 14 19:46:43.441685 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.441658 2557 generic.go:356] "Generic (PLEG): container finished" podID="c736c89e-7b1b-4d96-b36e-60956aca58af" containerID="1bc46a0e2b4f6c84376f807930e8f7d4d33434cb6fea0c6ce9eec135f31637a9" exitCode=0 Jul 14 19:46:43.441791 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.441739 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pdjn9" event={"ID":"c736c89e-7b1b-4d96-b36e-60956aca58af","Type":"ContainerDied","Data":"1bc46a0e2b4f6c84376f807930e8f7d4d33434cb6fea0c6ce9eec135f31637a9"} Jul 14 19:46:43.443222 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.443199 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" event={"ID":"87dcc8db-8293-45b1-bb67-1db76191f899","Type":"ContainerStarted","Data":"3aea1fe731229003b1d8d073241e98b2b3a95fdcd01055f8b1166e064304448e"} Jul 14 19:46:43.445994 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.445976 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 19:46:43.446297 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.446280 2557 generic.go:356] "Generic (PLEG): container finished" podID="15d09c81-aa45-4b25-83c2-c13f788e021c" containerID="40bc0498c97c4ca3192ba8eacb53bcd1bbf359281ea7de983858381c7d5c2c94" exitCode=1 Jul 14 19:46:43.446350 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.446339 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" event={"ID":"15d09c81-aa45-4b25-83c2-c13f788e021c","Type":"ContainerStarted","Data":"ae530f089454805d944cb23f61868115472586da9fcb0b0235793b264c93bc20"} Jul 14 19:46:43.446426 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.446406 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" event={"ID":"15d09c81-aa45-4b25-83c2-c13f788e021c","Type":"ContainerStarted","Data":"7e4503d29d33d8ac6c8ec820b7b90d480a786334476da9f7f0a378ce62101dd6"} Jul 14 19:46:43.446471 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.446463 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" event={"ID":"15d09c81-aa45-4b25-83c2-c13f788e021c","Type":"ContainerStarted","Data":"93698d612e9591597eca3d49672afc91679622d956055fafc54891d37bc70a4e"} Jul 14 19:46:43.446531 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.446508 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" event={"ID":"15d09c81-aa45-4b25-83c2-c13f788e021c","Type":"ContainerStarted","Data":"f42769b1130967815ff87347ee1538a92fc9692eea84337090d4f2b6e7f84775"} Jul 14 19:46:43.446582 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.446553 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" event={"ID":"15d09c81-aa45-4b25-83c2-c13f788e021c","Type":"ContainerDied","Data":"40bc0498c97c4ca3192ba8eacb53bcd1bbf359281ea7de983858381c7d5c2c94"} Jul 14 19:46:43.446627 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.446588 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" event={"ID":"15d09c81-aa45-4b25-83c2-c13f788e021c","Type":"ContainerStarted","Data":"20d6c771921c135781f81c4f05d2f156c9afb605aa1942324663ba51809b4534"} Jul 14 19:46:43.447666 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.447647 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rqfrv" event={"ID":"0e58d2f7-1d92-4bb0-aadd-83b5ecb40331","Type":"ContainerStarted","Data":"58df020b9ad9dadaea6a8291f3811d977126981ee0e05af7ad4cd4560be2e363"} Jul 14 19:46:43.448898 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.448881 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-knrrs" event={"ID":"839de40a-5fee-44af-8173-a65990f845cf","Type":"ContainerStarted","Data":"ca554ee72668ddc868e423ad7ef6e006d7f1992c4792696ed84b82f32b42719b"} Jul 14 19:46:43.450021 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.450000 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lpvwr" event={"ID":"fd7db434-46fc-4a27-83b2-acaefc0bd727","Type":"ContainerStarted","Data":"67804efe32b717a3478e18dd4a896e74636ad1f4993a50bd73df3b8dab279b0e"} Jul 14 19:46:43.451325 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.451304 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" event={"ID":"548df819-809f-4a00-bc11-2758f97617ed","Type":"ContainerStarted","Data":"556406d7e3e3f18700d8eff5bc1d0406b0c12522a0527628188e5bb2cfa2bea5"} Jul 14 19:46:43.452555 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.452534 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ff7s5" event={"ID":"322d4ffe-4783-4a5a-a18f-47878e117000","Type":"ContainerStarted","Data":"c72395abf8064bf6b54f5ed1a555d064fa441ba2760681d2da10a9117ce99ea7"} Jul 14 19:46:43.453803 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.453781 2557 generic.go:356] "Generic (PLEG): container finished" podID="4baf446df5fec806f69a038fd9e73461" containerID="9de5b318949f895458f72f667325b4e60ac197213bdfa56f31e797115f580e0c" exitCode=0 Jul 14 19:46:43.453895 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.453812 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal" event={"ID":"4baf446df5fec806f69a038fd9e73461","Type":"ContainerDied","Data":"9de5b318949f895458f72f667325b4e60ac197213bdfa56f31e797115f580e0c"} Jul 14 19:46:43.472826 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.472775 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rqfrv" podStartSLOduration=2.691218567 podStartE2EDuration="21.472758101s" totalImagesPullingTime="18.781539534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:43.472662001 +0000 UTC m=+21.552407795" watchObservedRunningTime="2026-07-14 19:46:43.472758101 +0000 UTC m=+21.552503895" Jul 14 19:46:43.473489 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.473443 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-184.ec2.internal" podStartSLOduration=20.473433413 podStartE2EDuration="20.473433413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:43.45451411 +0000 UTC m=+21.534259916" watchObservedRunningTime="2026-07-14 19:46:43.473433413 +0000 UTC m=+21.553179206" Jul 14 19:46:43.488344 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.488307 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ff7s5" podStartSLOduration=8.619990647 podStartE2EDuration="21.488289895s" totalImagesPullingTime="12.868299248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:43.487863751 +0000 UTC m=+21.567609544" watchObservedRunningTime="2026-07-14 19:46:43.488289895 +0000 UTC m=+21.568035738" Jul 14 19:46:43.516622 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.516581 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lpvwr" podStartSLOduration=8.657477495 podStartE2EDuration="21.516568278s" totalImagesPullingTime="12.859090783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:43.516496073 +0000 UTC m=+21.596242090" watchObservedRunningTime="2026-07-14 19:46:43.516568278 +0000 UTC m=+21.596314069" Jul 14 19:46:43.534017 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.533982 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d7bzh" podStartSLOduration=2.820522294 podStartE2EDuration="21.533969001s" totalImagesPullingTime="18.713446707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:43.533746091 +0000 UTC m=+21.613491883" watchObservedRunningTime="2026-07-14 19:46:43.533969001 +0000 UTC m=+21.613714853" Jul 14 19:46:43.548414 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:43.548373 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-knrrs" podStartSLOduration=2.861387918 podStartE2EDuration="21.548359495s" totalImagesPullingTime="18.686971577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:43.548073678 +0000 UTC m=+21.627819471" watchObservedRunningTime="2026-07-14 19:46:43.548359495 +0000 UTC m=+21.628105287" Jul 14 19:46:44.358078 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:44.358026 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:44.358361 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:44.358158 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:44.358361 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:44.358204 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:44.358361 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:44.358308 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:44.456870 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:44.456839 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ck7z5" event={"ID":"cfcbd9c8-4dbd-4ebc-b0be-0ee1f2e70378","Type":"ContainerStarted","Data":"4c719fc70e2e3357cf627f71e86d1ae8eed7d7c0043f314dc396523d35e35e29"} Jul 14 19:46:44.458576 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:44.458545 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal" event={"ID":"4baf446df5fec806f69a038fd9e73461","Type":"ContainerStarted","Data":"cec7463b30eabdff156431f932aae24fe5b84a6ecc449c039b4a74b28437cbdc"} Jul 14 19:46:44.489289 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:44.489244 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-184.ec2.internal" podStartSLOduration=21.489232232 podStartE2EDuration="21.489232232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:44.488202721 +0000 UTC m=+22.567948512" watchObservedRunningTime="2026-07-14 19:46:44.489232232 +0000 UTC m=+22.568978023" Jul 14 19:46:44.489663 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:44.489637 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ck7z5" podStartSLOduration=3.768453595 podStartE2EDuration="22.489630145s" totalImagesPullingTime="18.72117655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:44.473268914 +0000 UTC m=+22.553014699" watchObservedRunningTime="2026-07-14 19:46:44.489630145 +0000 UTC m=+22.569375936" Jul 14 19:46:45.136947 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:45.136904 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ff7s5" Jul 14 19:46:45.137466 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:45.137441 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ff7s5" Jul 14 19:46:45.355013 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:45.354991 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:45.355156 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:45.355131 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:45.388837 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:45.388802 2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 19:46:45.463288 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:45.463251 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" event={"ID":"548df819-809f-4a00-bc11-2758f97617ed","Type":"ContainerStarted","Data":"17904170be076a23259a585a873e52c862b85cb8f994cbda758c56b9ddbcb970"} Jul 14 19:46:45.466827 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:45.466806 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 19:46:45.467284 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:45.467263 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" event={"ID":"15d09c81-aa45-4b25-83c2-c13f788e021c","Type":"ContainerStarted","Data":"7f7fa6d5d60c1ac7bcb26125b903370ae1ff1d3a16c5b00c6c33935cca3f78a4"} Jul 14 19:46:45.467821 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:45.467695 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ff7s5" Jul 14 19:46:45.468103 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:45.468086 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ff7s5" Jul 14 19:46:45.819514 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:45.819438 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lpvwr_fd7db434-46fc-4a27-83b2-acaefc0bd727/dns-node-resolver/0.log" Jul 14 19:46:46.319037 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:46.318682 2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T19:46:45.388820015Z","UUID":"fcc57ba9-9379-4312-b8b8-8129110a0bdd","Handler":null,"Name":"","Endpoint":""} Jul 14 19:46:46.320714 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:46.320688 2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 19:46:46.320838 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:46.320753 2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 19:46:46.354706 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:46.354678 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:46.354706 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:46.354707 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:46.354940 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:46.354799 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:46.354940 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:46.354873 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:46.981669 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:46.981646 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-knrrs_839de40a-5fee-44af-8173-a65990f845cf/node-ca/0.log" Jul 14 19:46:47.355233 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:47.355199 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:47.355408 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:47.355319 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:48.357378 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:48.357345 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:48.357911 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:48.357486 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:48.357911 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:48.357527 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:48.357911 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:48.357604 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:49.355551 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:49.355375 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:49.355714 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:49.355626 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:49.478801 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:49.478764 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" event={"ID":"548df819-809f-4a00-bc11-2758f97617ed","Type":"ContainerStarted","Data":"0d1598d0f896878e9fb1030e05f4152a3482b89038567c7e39a5a3dca193bb6c"} Jul 14 19:46:49.480824 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:49.480799 2557 generic.go:356] "Generic (PLEG): container finished" podID="c736c89e-7b1b-4d96-b36e-60956aca58af" containerID="0ec07c40b573de3878e071a29c7e279fc242bc5e34d7ce3046b623196a6d2b9a" exitCode=0 Jul 14 19:46:49.481012 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:49.480885 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pdjn9" event={"ID":"c736c89e-7b1b-4d96-b36e-60956aca58af","Type":"ContainerDied","Data":"0ec07c40b573de3878e071a29c7e279fc242bc5e34d7ce3046b623196a6d2b9a"} Jul 14 19:46:49.484583 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:49.484568 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 19:46:49.484871 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:49.484854 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" event={"ID":"15d09c81-aa45-4b25-83c2-c13f788e021c","Type":"ContainerStarted","Data":"e96a8b89135657e1a898b8ff8409d7d3c8ed34ca76a8c2b2650f10128528871b"} Jul 14 19:46:49.485108 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:49.485090 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:49.485185 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:49.485163 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:49.485234 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:49.485212 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:49.485281 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:49.485235 2557 scope.go:117] "RemoveContainer" containerID="40bc0498c97c4ca3192ba8eacb53bcd1bbf359281ea7de983858381c7d5c2c94" Jul 14 19:46:49.501150 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:49.501128 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:49.501376 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:49.501361 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:46:49.502157 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:49.502084 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2w8r" podStartSLOduration=2.923960825 podStartE2EDuration="27.502071096s" totalImagesPullingTime="24.578110271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:49.501948431 +0000 UTC m=+27.581694233" watchObservedRunningTime="2026-07-14 19:46:49.502071096 +0000 UTC m=+27.581816878" Jul 14 19:46:50.355431 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:50.355398 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:50.355649 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:50.355521 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:50.355844 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:50.355412 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:50.355990 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:50.355950 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:50.490540 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:50.490520 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 19:46:50.490948 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:50.490874 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" event={"ID":"15d09c81-aa45-4b25-83c2-c13f788e021c","Type":"ContainerStarted","Data":"dc3bbd8ee7c69da911c4de5bf90d37fd7c5994c78f31067b8993b335c1333b77"} Jul 14 19:46:50.524181 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:50.524142 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" podStartSLOduration=9.743420114 podStartE2EDuration="28.524127132s" totalImagesPullingTime="18.780707018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:50.523007482 +0000 UTC m=+28.602753311" watchObservedRunningTime="2026-07-14 19:46:50.524127132 +0000 UTC m=+28.603872924" Jul 14 19:46:51.355144 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:51.355111 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:51.355342 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:51.355227 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:51.495029 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:51.494999 2557 generic.go:356] "Generic (PLEG): container finished" podID="c736c89e-7b1b-4d96-b36e-60956aca58af" containerID="4d40a2f8eba770a2badd864a46a64044d68a402eb1caa589c76a24bbe5c1c933" exitCode=0 Jul 14 19:46:51.495417 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:51.495089 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pdjn9" event={"ID":"c736c89e-7b1b-4d96-b36e-60956aca58af","Type":"ContainerDied","Data":"4d40a2f8eba770a2badd864a46a64044d68a402eb1caa589c76a24bbe5c1c933"} Jul 14 19:46:52.356102 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:52.356028 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:52.356275 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:52.356141 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:52.356275 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:52.356157 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:52.356387 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:52.356268 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:53.355248 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:53.355207 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:53.355636 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:53.355336 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:53.502706 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:53.502663 2557 generic.go:356] "Generic (PLEG): container finished" podID="c736c89e-7b1b-4d96-b36e-60956aca58af" containerID="c0d7eafb25c7c658ae8edb6c263c4f7261291a3d8d2c05d2e544b681eba567b7" exitCode=0 Jul 14 19:46:53.502853 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:53.502728 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pdjn9" event={"ID":"c736c89e-7b1b-4d96-b36e-60956aca58af","Type":"ContainerDied","Data":"c0d7eafb25c7c658ae8edb6c263c4f7261291a3d8d2c05d2e544b681eba567b7"} Jul 14 19:46:54.354866 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:54.354832 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:54.355044 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:54.354830 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:54.355044 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:54.354980 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:54.355191 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:54.355126 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:55.100184 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:55.100149 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs\") pod \"network-metrics-daemon-bh66q\" (UID: \"a2c6f5f5-7f01-421d-bb94-170b7a1cda67\") " pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:55.100621 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:55.100208 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:55.100621 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:55.100326 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:55.100621 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:55.100400 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs podName:a2c6f5f5-7f01-421d-bb94-170b7a1cda67 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:27.100379802 +0000 UTC m=+65.180125589 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs") pod "network-metrics-daemon-bh66q" (UID: "a2c6f5f5-7f01-421d-bb94-170b7a1cda67") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:55.100621 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:55.100336 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:55.100621 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:55.100470 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret podName:d3f090a3-021a-4185-bcc8-7c00d7d56b77 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:27.100454201 +0000 UTC m=+65.180199980 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret") pod "global-pull-secret-syncer-6w8gx" (UID: "d3f090a3-021a-4185-bcc8-7c00d7d56b77") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:55.201528 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:55.201487 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhw2\" (UniqueName: \"kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2\") pod \"network-check-target-xppkv\" (UID: \"b0396d27-4305-4393-8d0e-f9a5cff12337\") " pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:55.201710 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:55.201665 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:55.201710 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:55.201689 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:55.201710 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:55.201701 2557 projected.go:196] Error preparing data for projected volume kube-api-access-qmhw2 for pod openshift-network-diagnostics/network-check-target-xppkv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:55.201871 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:55.201754 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2 podName:b0396d27-4305-4393-8d0e-f9a5cff12337 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:27.201740214 +0000 UTC m=+65.281485984 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmhw2" (UniqueName: "kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2") pod "network-check-target-xppkv" (UID: "b0396d27-4305-4393-8d0e-f9a5cff12337") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:55.355645 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:55.355565 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:55.355792 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:55.355680 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:56.355556 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:56.355517 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:56.356009 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:56.355661 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:56.356009 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:56.355700 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:56.356009 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:56.355781 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:57.354798 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:57.354763 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:57.354975 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:57.354867 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:58.354711 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:58.354685 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:46:58.355087 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:58.354724 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:46:58.355087 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:58.354800 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:46:58.355087 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:58.354941 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:46:58.517219 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:58.517185 2557 generic.go:356] "Generic (PLEG): container finished" podID="c736c89e-7b1b-4d96-b36e-60956aca58af" containerID="1470762a1126c6bbcce6dcd2730f39c94b9eaf6fa8fdb2ba2ec7632432d38395" exitCode=0 Jul 14 19:46:58.517391 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:58.517247 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pdjn9" event={"ID":"c736c89e-7b1b-4d96-b36e-60956aca58af","Type":"ContainerDied","Data":"1470762a1126c6bbcce6dcd2730f39c94b9eaf6fa8fdb2ba2ec7632432d38395"} Jul 14 19:46:59.355310 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:59.355282 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:46:59.355690 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:46:59.355368 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:46:59.522370 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:59.522339 2557 generic.go:356] "Generic (PLEG): container finished" podID="c736c89e-7b1b-4d96-b36e-60956aca58af" containerID="5b3c54d9e0626ac61d54de8218f0c9378ae8b87a787a64b188c6c7200b68c87d" exitCode=0 Jul 14 19:46:59.522521 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:46:59.522393 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pdjn9" event={"ID":"c736c89e-7b1b-4d96-b36e-60956aca58af","Type":"ContainerDied","Data":"5b3c54d9e0626ac61d54de8218f0c9378ae8b87a787a64b188c6c7200b68c87d"} Jul 14 19:47:00.354884 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:00.354845 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:47:00.355074 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:00.354958 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:47:00.355074 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:00.355019 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:47:00.355169 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:00.355146 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:47:00.527991 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:00.527954 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pdjn9" event={"ID":"c736c89e-7b1b-4d96-b36e-60956aca58af","Type":"ContainerStarted","Data":"e70595bc94e01a931fb56ca4270e0341a8951b9c9317b660b89dc639733ff41b"} Jul 14 19:47:00.549242 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:00.549202 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pdjn9" podStartSLOduration=8.120362516 podStartE2EDuration="38.549187478s" totalImagesPullingTime="30.428824962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:00.548203784 +0000 UTC m=+38.627949576" watchObservedRunningTime="2026-07-14 19:47:00.549187478 +0000 UTC m=+38.628933269" Jul 14 19:47:01.354979 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:01.354944 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:47:01.355201 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:01.355075 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:47:02.355392 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:02.355366 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:47:02.355801 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:02.355440 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:47:02.355801 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:02.355454 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:47:02.356034 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:02.356003 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:47:03.355029 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:03.355002 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:47:03.355216 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:03.355112 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:47:04.355404 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:04.355372 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:47:04.355764 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:04.355479 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:47:04.355764 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:04.355535 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:47:04.355764 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:04.355614 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:47:05.355473 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:05.355443 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:47:05.355830 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:05.355535 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:47:06.355618 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:06.355581 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:47:06.356077 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:06.355709 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:47:06.356077 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:06.355754 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:47:06.356077 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:06.355840 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:47:07.355586 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:07.355555 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:47:07.355768 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:07.355655 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:47:08.002361 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:08.002330 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bh66q"] Jul 14 19:47:08.002530 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:08.002458 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:47:08.002591 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:08.002544 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:47:08.005310 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:08.005281 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xppkv"] Jul 14 19:47:08.005435 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:08.005403 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:47:08.005543 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:08.005514 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:47:08.015507 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:08.015480 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6w8gx"] Jul 14 19:47:08.015640 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:08.015572 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:47:08.015681 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:08.015652 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:47:09.354779 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:09.354749 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:47:09.355450 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:09.354849 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:47:10.355378 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:10.355342 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:47:10.355809 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:10.355470 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6w8gx" podUID="d3f090a3-021a-4185-bcc8-7c00d7d56b77" Jul 14 19:47:10.355809 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:10.355519 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:47:10.355809 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:10.355616 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bh66q" podUID="a2c6f5f5-7f01-421d-bb94-170b7a1cda67" Jul 14 19:47:11.355113 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:11.355080 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:47:11.355270 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:11.355180 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xppkv" podUID="b0396d27-4305-4393-8d0e-f9a5cff12337" Jul 14 19:47:12.236708 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.236478 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-184.ec2.internal" event="NodeReady" Jul 14 19:47:12.237052 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.236729 2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 19:47:12.286216 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.286186 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-49ddf"] Jul 14 19:47:12.294756 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.294733 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-krtpl"] Jul 14 19:47:12.294883 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.294849 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-49ddf" Jul 14 19:47:12.297029 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.297005 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-46xb9\"" Jul 14 19:47:12.297222 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.297210 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 19:47:12.298821 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.298568 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 19:47:12.298821 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.298589 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 19:47:12.313582 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.313555 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-49ddf"] Jul 14 19:47:12.313690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.313593 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-krtpl"] Jul 14 19:47:12.313690 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.313672 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.313786 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.313755 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9fb96"] Jul 14 19:47:12.316850 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.316829 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 19:47:12.316998 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.316830 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 19:47:12.317086 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.317071 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 19:47:12.317207 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.317187 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 19:47:12.317431 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.317396 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bpj6w\"" Jul 14 19:47:12.325216 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.325195 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9fb96"] Jul 14 19:47:12.325323 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.325306 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:12.327790 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.327765 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 19:47:12.327887 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.327794 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7w2mk\"" Jul 14 19:47:12.327887 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.327868 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 19:47:12.327981 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.327885 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af5e2ec9-a131-4164-b80e-3045f199612b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krtpl\" (UID: \"af5e2ec9-a131-4164-b80e-3045f199612b\") " pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.327981 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.327910 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a7dce1a-0e21-4d14-a65e-2e2b28d69f70-cert\") pod \"ingress-canary-49ddf\" (UID: \"4a7dce1a-0e21-4d14-a65e-2e2b28d69f70\") " pod="openshift-ingress-canary/ingress-canary-49ddf" Jul 14 19:47:12.327981 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.327934 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs8lr\" (UniqueName: \"kubernetes.io/projected/4a7dce1a-0e21-4d14-a65e-2e2b28d69f70-kube-api-access-rs8lr\") pod \"ingress-canary-49ddf\" (UID: \"4a7dce1a-0e21-4d14-a65e-2e2b28d69f70\") " pod="openshift-ingress-canary/ingress-canary-49ddf" Jul 14 19:47:12.328155 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.327996 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06b84d98-4dc1-47d4-8330-b292e14940f1-config-volume\") pod \"dns-default-9fb96\" (UID: \"06b84d98-4dc1-47d4-8330-b292e14940f1\") " pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:12.328155 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.328070 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06b84d98-4dc1-47d4-8330-b292e14940f1-tmp-dir\") pod \"dns-default-9fb96\" (UID: \"06b84d98-4dc1-47d4-8330-b292e14940f1\") " pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:12.328155 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.328107 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvkm5\" (UniqueName: \"kubernetes.io/projected/af5e2ec9-a131-4164-b80e-3045f199612b-kube-api-access-gvkm5\") pod \"insights-runtime-extractor-krtpl\" (UID: \"af5e2ec9-a131-4164-b80e-3045f199612b\") " pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.328155 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.328145 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af5e2ec9-a131-4164-b80e-3045f199612b-crio-socket\") pod \"insights-runtime-extractor-krtpl\" (UID: \"af5e2ec9-a131-4164-b80e-3045f199612b\") " pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.328341 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.328168 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af5e2ec9-a131-4164-b80e-3045f199612b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krtpl\" (UID: \"af5e2ec9-a131-4164-b80e-3045f199612b\") " pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.328341 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.328218 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06b84d98-4dc1-47d4-8330-b292e14940f1-metrics-tls\") pod \"dns-default-9fb96\" (UID: \"06b84d98-4dc1-47d4-8330-b292e14940f1\") " pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:12.328341 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.328241 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlft2\" (UniqueName: \"kubernetes.io/projected/06b84d98-4dc1-47d4-8330-b292e14940f1-kube-api-access-mlft2\") pod \"dns-default-9fb96\" (UID: \"06b84d98-4dc1-47d4-8330-b292e14940f1\") " pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:12.328341 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.328276 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af5e2ec9-a131-4164-b80e-3045f199612b-data-volume\") pod \"insights-runtime-extractor-krtpl\" (UID: \"af5e2ec9-a131-4164-b80e-3045f199612b\") " pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.354904 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.354875 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:47:12.355018 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.354881 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:47:12.358377 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.358357 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 19:47:12.359173 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.359102 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 19:47:12.359516 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.359247 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mvlkh\"" Jul 14 19:47:12.428884 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.428858 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06b84d98-4dc1-47d4-8330-b292e14940f1-metrics-tls\") pod \"dns-default-9fb96\" (UID: \"06b84d98-4dc1-47d4-8330-b292e14940f1\") " pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:12.428884 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.428886 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlft2\" (UniqueName: \"kubernetes.io/projected/06b84d98-4dc1-47d4-8330-b292e14940f1-kube-api-access-mlft2\") pod \"dns-default-9fb96\" (UID: \"06b84d98-4dc1-47d4-8330-b292e14940f1\") " pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:12.429100 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.428912 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af5e2ec9-a131-4164-b80e-3045f199612b-data-volume\") pod \"insights-runtime-extractor-krtpl\" (UID: \"af5e2ec9-a131-4164-b80e-3045f199612b\") " pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.429189 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.429147 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af5e2ec9-a131-4164-b80e-3045f199612b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krtpl\" (UID: \"af5e2ec9-a131-4164-b80e-3045f199612b\") " pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.429254 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.429188 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a7dce1a-0e21-4d14-a65e-2e2b28d69f70-cert\") pod \"ingress-canary-49ddf\" (UID: \"4a7dce1a-0e21-4d14-a65e-2e2b28d69f70\") " pod="openshift-ingress-canary/ingress-canary-49ddf" Jul 14 19:47:12.429254 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.429212 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af5e2ec9-a131-4164-b80e-3045f199612b-data-volume\") pod \"insights-runtime-extractor-krtpl\" (UID: \"af5e2ec9-a131-4164-b80e-3045f199612b\") " pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.429254 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.429217 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs8lr\" (UniqueName: \"kubernetes.io/projected/4a7dce1a-0e21-4d14-a65e-2e2b28d69f70-kube-api-access-rs8lr\") pod \"ingress-canary-49ddf\" (UID: \"4a7dce1a-0e21-4d14-a65e-2e2b28d69f70\") " pod="openshift-ingress-canary/ingress-canary-49ddf" Jul 14 19:47:12.429401 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.429254 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06b84d98-4dc1-47d4-8330-b292e14940f1-config-volume\") pod \"dns-default-9fb96\" (UID: \"06b84d98-4dc1-47d4-8330-b292e14940f1\") " pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:12.429401 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.429281 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06b84d98-4dc1-47d4-8330-b292e14940f1-tmp-dir\") pod \"dns-default-9fb96\" (UID: \"06b84d98-4dc1-47d4-8330-b292e14940f1\") " pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:12.429401 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.429306 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvkm5\" (UniqueName: \"kubernetes.io/projected/af5e2ec9-a131-4164-b80e-3045f199612b-kube-api-access-gvkm5\") pod \"insights-runtime-extractor-krtpl\" (UID: \"af5e2ec9-a131-4164-b80e-3045f199612b\") " pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.429401 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.429348 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af5e2ec9-a131-4164-b80e-3045f199612b-crio-socket\") pod \"insights-runtime-extractor-krtpl\" (UID: \"af5e2ec9-a131-4164-b80e-3045f199612b\") " pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.429401 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.429374 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af5e2ec9-a131-4164-b80e-3045f199612b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krtpl\" (UID: \"af5e2ec9-a131-4164-b80e-3045f199612b\") " pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.429703 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.429671 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06b84d98-4dc1-47d4-8330-b292e14940f1-tmp-dir\") pod \"dns-default-9fb96\" (UID: \"06b84d98-4dc1-47d4-8330-b292e14940f1\") " pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:12.430075 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.430037 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af5e2ec9-a131-4164-b80e-3045f199612b-crio-socket\") pod \"insights-runtime-extractor-krtpl\" (UID: \"af5e2ec9-a131-4164-b80e-3045f199612b\") " pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.431008 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.430985 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af5e2ec9-a131-4164-b80e-3045f199612b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krtpl\" (UID: \"af5e2ec9-a131-4164-b80e-3045f199612b\") " pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.433023 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.433003 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af5e2ec9-a131-4164-b80e-3045f199612b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krtpl\" (UID: \"af5e2ec9-a131-4164-b80e-3045f199612b\") " pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.433109 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.433003 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06b84d98-4dc1-47d4-8330-b292e14940f1-metrics-tls\") pod \"dns-default-9fb96\" (UID: \"06b84d98-4dc1-47d4-8330-b292e14940f1\") " pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:12.433157 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.433111 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a7dce1a-0e21-4d14-a65e-2e2b28d69f70-cert\") pod \"ingress-canary-49ddf\" (UID: \"4a7dce1a-0e21-4d14-a65e-2e2b28d69f70\") " pod="openshift-ingress-canary/ingress-canary-49ddf" Jul 14 19:47:12.438911 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.438138 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06b84d98-4dc1-47d4-8330-b292e14940f1-config-volume\") pod \"dns-default-9fb96\" (UID: \"06b84d98-4dc1-47d4-8330-b292e14940f1\") " pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:12.440391 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.440370 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlft2\" (UniqueName: \"kubernetes.io/projected/06b84d98-4dc1-47d4-8330-b292e14940f1-kube-api-access-mlft2\") pod \"dns-default-9fb96\" (UID: \"06b84d98-4dc1-47d4-8330-b292e14940f1\") " pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:12.440498 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.440472 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvkm5\" (UniqueName: \"kubernetes.io/projected/af5e2ec9-a131-4164-b80e-3045f199612b-kube-api-access-gvkm5\") pod \"insights-runtime-extractor-krtpl\" (UID: \"af5e2ec9-a131-4164-b80e-3045f199612b\") " pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.441121 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.441103 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs8lr\" (UniqueName: \"kubernetes.io/projected/4a7dce1a-0e21-4d14-a65e-2e2b28d69f70-kube-api-access-rs8lr\") pod \"ingress-canary-49ddf\" (UID: \"4a7dce1a-0e21-4d14-a65e-2e2b28d69f70\") " pod="openshift-ingress-canary/ingress-canary-49ddf" Jul 14 19:47:12.605857 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.605824 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-49ddf" Jul 14 19:47:12.624116 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.624067 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-krtpl" Jul 14 19:47:12.633488 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.633466 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:12.780517 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.780486 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-krtpl"] Jul 14 19:47:12.783165 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.782939 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9fb96"] Jul 14 19:47:12.787956 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:12.787924 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-49ddf"] Jul 14 19:47:12.792632 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:47:12.792610 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7dce1a_0e21_4d14_a65e_2e2b28d69f70.slice/crio-aff65d2f96a7d0cb0b53bdc53a2f3be2d9c842799668f74168140f21572c2ab0 WatchSource:0}: Error finding container aff65d2f96a7d0cb0b53bdc53a2f3be2d9c842799668f74168140f21572c2ab0: Status 404 returned error can't find the container with id aff65d2f96a7d0cb0b53bdc53a2f3be2d9c842799668f74168140f21572c2ab0 Jul 14 19:47:13.355229 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:13.355186 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:47:13.357623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:13.357602 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xm9xl\"" Jul 14 19:47:13.357715 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:13.357603 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 19:47:13.358370 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:13.358354 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 19:47:13.558466 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:13.558429 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krtpl" event={"ID":"af5e2ec9-a131-4164-b80e-3045f199612b","Type":"ContainerStarted","Data":"cc21da853846aa0ec9511892304e04544c7fa938d6ab9f5c35933b7bae09d27e"} Jul 14 19:47:13.559958 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:13.559900 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-49ddf" event={"ID":"4a7dce1a-0e21-4d14-a65e-2e2b28d69f70","Type":"ContainerStarted","Data":"aff65d2f96a7d0cb0b53bdc53a2f3be2d9c842799668f74168140f21572c2ab0"} Jul 14 19:47:13.561393 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:13.561333 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9fb96" event={"ID":"06b84d98-4dc1-47d4-8330-b292e14940f1","Type":"ContainerStarted","Data":"d41eef467485184430b7583aaeaaa23c11c94efbc87182fac31c2629b1ed356a"} Jul 14 19:47:16.026908 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.026691 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59fb8745d5-wb5dk"] Jul 14 19:47:16.037799 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.037775 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.040260 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.040198 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59fb8745d5-wb5dk"] Jul 14 19:47:16.040654 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.040534 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 19:47:16.040654 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.040544 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 19:47:16.040654 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.040614 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 19:47:16.040654 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.040639 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h5xm7\"" Jul 14 19:47:16.041078 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.040916 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 19:47:16.041751 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.041559 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 19:47:16.041751 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.041608 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 19:47:16.041751 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.041563 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 19:47:16.057466 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.057443 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-config\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.057597 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.057518 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-serving-cert\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.057597 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.057537 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-service-ca\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.057597 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.057553 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk96v\" (UniqueName: \"kubernetes.io/projected/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-kube-api-access-qk96v\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.057597 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.057585 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-oauth-config\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.057806 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.057614 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-oauth-serving-cert\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.158625 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.158578 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-oauth-config\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.158804 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.158634 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-oauth-serving-cert\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.158804 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.158696 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-config\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.158804 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.158776 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-serving-cert\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.158960 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.158805 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-service-ca\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.158960 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.158829 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk96v\" (UniqueName: \"kubernetes.io/projected/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-kube-api-access-qk96v\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.159547 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.159524 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-oauth-serving-cert\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.159698 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.159552 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-service-ca\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.159698 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.159524 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-config\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.163107 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.163087 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-serving-cert\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.163107 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.163102 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-oauth-config\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.167706 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.167663 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk96v\" (UniqueName: \"kubernetes.io/projected/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-kube-api-access-qk96v\") pod \"console-59fb8745d5-wb5dk\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:16.350603 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:16.350556 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:18.302171 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:18.302141 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59fb8745d5-wb5dk"] Jul 14 19:47:18.305195 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:47:18.305170 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719cb6c1_7370_4e96_bd1a_6701eec6bb6a.slice/crio-fecc3959dc3442532fa49c15a2d9d403f699e36db677078ddc6e015074f784e2 WatchSource:0}: Error finding container fecc3959dc3442532fa49c15a2d9d403f699e36db677078ddc6e015074f784e2: Status 404 returned error can't find the container with id fecc3959dc3442532fa49c15a2d9d403f699e36db677078ddc6e015074f784e2 Jul 14 19:47:18.579559 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:18.579529 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9fb96" event={"ID":"06b84d98-4dc1-47d4-8330-b292e14940f1","Type":"ContainerStarted","Data":"54a5bd300912e1c37b8e4e5559a19101966ef1ac89e379ba0bd87338533bc3b7"} Jul 14 19:47:18.579678 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:18.579591 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9fb96" event={"ID":"06b84d98-4dc1-47d4-8330-b292e14940f1","Type":"ContainerStarted","Data":"b0c6527f9bc6de51d345870247116f47b7a920710ddf7d945d4a897e1c9d150c"} Jul 14 19:47:18.581022 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:18.581001 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krtpl" event={"ID":"af5e2ec9-a131-4164-b80e-3045f199612b","Type":"ContainerStarted","Data":"cdf3d172ba4019aa578b350090618c361553f660d11c2f63721418e312083474"} Jul 14 19:47:18.582329 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:18.582301 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59fb8745d5-wb5dk" event={"ID":"719cb6c1-7370-4e96-bd1a-6701eec6bb6a","Type":"ContainerStarted","Data":"fecc3959dc3442532fa49c15a2d9d403f699e36db677078ddc6e015074f784e2"} Jul 14 19:47:18.583820 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:18.583799 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-49ddf" event={"ID":"4a7dce1a-0e21-4d14-a65e-2e2b28d69f70","Type":"ContainerStarted","Data":"cc2bbc8c29a15813b0eaa3f5cbb5e407ae118b26c16f0cd6ccbfa6cf3cf654ad"} Jul 14 19:47:18.600518 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:18.600455 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-49ddf" podStartSLOduration=1.220494383 podStartE2EDuration="6.600435987s" totalImagesPullingTime="5.379941604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:18.599082908 +0000 UTC m=+56.678828719" watchObservedRunningTime="2026-07-14 19:47:18.600435987 +0000 UTC m=+56.680181781" Jul 14 19:47:19.608310 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:19.608280 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:19.627228 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:19.627174 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9fb96" podStartSLOduration=2.242532478 podStartE2EDuration="7.627155784s" totalImagesPullingTime="5.384623306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:19.625810035 +0000 UTC m=+57.705555845" watchObservedRunningTime="2026-07-14 19:47:19.627155784 +0000 UTC m=+57.706901617" Jul 14 19:47:20.591475 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:20.591436 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krtpl" event={"ID":"af5e2ec9-a131-4164-b80e-3045f199612b","Type":"ContainerStarted","Data":"69f02f26b3bb1693b33725c537317f29a011e737a1d18ea27f01be1cc8ed445b"} Jul 14 19:47:21.508603 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:21.508572 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-926lj" Jul 14 19:47:22.055474 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.055436 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5888c58fc-qzxlt"] Jul 14 19:47:22.057259 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.057236 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.064849 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.064820 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 14 19:47:22.071267 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.071240 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5888c58fc-qzxlt"] Jul 14 19:47:22.104687 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.104664 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-service-ca\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.104842 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.104694 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7742d52-e657-447d-a209-61d317713990-console-oauth-config\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.104842 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.104719 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6bdp\" (UniqueName: \"kubernetes.io/projected/d7742d52-e657-447d-a209-61d317713990-kube-api-access-v6bdp\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.104842 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.104755 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7742d52-e657-447d-a209-61d317713990-console-serving-cert\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.104842 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.104781 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-trusted-ca-bundle\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.104842 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.104835 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-console-config\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.105041 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.104884 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-oauth-serving-cert\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.205249 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.205220 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-oauth-serving-cert\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.205444 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.205270 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-service-ca\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.205444 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.205299 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7742d52-e657-447d-a209-61d317713990-console-oauth-config\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.205444 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.205330 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6bdp\" (UniqueName: \"kubernetes.io/projected/d7742d52-e657-447d-a209-61d317713990-kube-api-access-v6bdp\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.205444 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.205361 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7742d52-e657-447d-a209-61d317713990-console-serving-cert\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.205444 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.205384 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-trusted-ca-bundle\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.205708 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.205458 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-console-config\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.206267 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.206029 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-service-ca\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.206267 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.206073 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-oauth-serving-cert\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.206267 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.206217 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-console-config\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.206601 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.206580 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-trusted-ca-bundle\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.208097 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.208049 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7742d52-e657-447d-a209-61d317713990-console-oauth-config\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.208257 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.208241 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7742d52-e657-447d-a209-61d317713990-console-serving-cert\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.216564 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.216539 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6bdp\" (UniqueName: \"kubernetes.io/projected/d7742d52-e657-447d-a209-61d317713990-kube-api-access-v6bdp\") pod \"console-5888c58fc-qzxlt\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:22.367536 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:22.367509 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:23.263213 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:23.263184 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5888c58fc-qzxlt"] Jul 14 19:47:23.266163 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:47:23.266132 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7742d52_e657_447d_a209_61d317713990.slice/crio-293e34edb2656db2c170993928968f7bbcc03e86910dbb66d10d8675e1673a6a WatchSource:0}: Error finding container 293e34edb2656db2c170993928968f7bbcc03e86910dbb66d10d8675e1673a6a: Status 404 returned error can't find the container with id 293e34edb2656db2c170993928968f7bbcc03e86910dbb66d10d8675e1673a6a Jul 14 19:47:23.603106 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:23.603052 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krtpl" event={"ID":"af5e2ec9-a131-4164-b80e-3045f199612b","Type":"ContainerStarted","Data":"7a794138ad4666af8e7fc159ae8b300c6400fddc5274a3444025160eeef1aa92"} Jul 14 19:47:23.604618 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:23.604591 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59fb8745d5-wb5dk" event={"ID":"719cb6c1-7370-4e96-bd1a-6701eec6bb6a","Type":"ContainerStarted","Data":"e16a195d12cee5c54c2f0c3fb125149064801eefc9c8ba3b6b838ac9ea555ab2"} Jul 14 19:47:23.606112 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:23.606079 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5888c58fc-qzxlt" event={"ID":"d7742d52-e657-447d-a209-61d317713990","Type":"ContainerStarted","Data":"00a917726ebf827ec90b8ad05213c608f0258f0e1a0592ba89d2befee794f848"} Jul 14 19:47:23.606218 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:23.606121 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5888c58fc-qzxlt" event={"ID":"d7742d52-e657-447d-a209-61d317713990","Type":"ContainerStarted","Data":"293e34edb2656db2c170993928968f7bbcc03e86910dbb66d10d8675e1673a6a"} Jul 14 19:47:23.622207 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:23.622163 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-krtpl" podStartSLOduration=1.5110669909999999 podStartE2EDuration="11.622147095s" totalImagesPullingTime="10.111080104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:23.620917421 +0000 UTC m=+61.700663241" watchObservedRunningTime="2026-07-14 19:47:23.622147095 +0000 UTC m=+61.701892887" Jul 14 19:47:23.645453 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:23.645385 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59fb8745d5-wb5dk" podStartSLOduration=2.817880388 podStartE2EDuration="7.645365256s" totalImagesPullingTime="4.827484868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:23.644112471 +0000 UTC m=+61.723858282" watchObservedRunningTime="2026-07-14 19:47:23.645365256 +0000 UTC m=+61.725111049" Jul 14 19:47:23.661799 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:23.661741 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5888c58fc-qzxlt" podStartSLOduration=1.661723244 podStartE2EDuration="1.661723244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:23.661225795 +0000 UTC m=+61.740971632" watchObservedRunningTime="2026-07-14 19:47:23.661723244 +0000 UTC m=+61.741469037" Jul 14 19:47:26.351428 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:26.351395 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:26.351818 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:26.351450 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:26.352685 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:26.352662 2557 patch_prober.go:28] interesting pod/console-59fb8745d5-wb5dk container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused" start-of-body= Jul 14 19:47:26.352737 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:26.352704 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-59fb8745d5-wb5dk" podUID="719cb6c1-7370-4e96-bd1a-6701eec6bb6a" containerName="console" probeResult="failure" output="Get \"https://10.133.0.9:8443/health\": dial tcp 10.133.0.9:8443: connect: connection refused" Jul 14 19:47:27.141635 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.141600 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs\") pod \"network-metrics-daemon-bh66q\" (UID: \"a2c6f5f5-7f01-421d-bb94-170b7a1cda67\") " pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:47:27.141812 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.141647 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:47:27.144123 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.144101 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 19:47:27.144233 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.144104 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 19:47:27.154250 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.154229 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c6f5f5-7f01-421d-bb94-170b7a1cda67-metrics-certs\") pod \"network-metrics-daemon-bh66q\" (UID: \"a2c6f5f5-7f01-421d-bb94-170b7a1cda67\") " pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:47:27.154339 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.154318 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d3f090a3-021a-4185-bcc8-7c00d7d56b77-original-pull-secret\") pod \"global-pull-secret-syncer-6w8gx\" (UID: \"d3f090a3-021a-4185-bcc8-7c00d7d56b77\") " pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:47:27.242830 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.242794 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhw2\" (UniqueName: \"kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2\") pod \"network-check-target-xppkv\" (UID: \"b0396d27-4305-4393-8d0e-f9a5cff12337\") " pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:47:27.245540 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.245522 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 19:47:27.255345 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.255325 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 19:47:27.266284 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.266263 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmhw2\" (UniqueName: \"kubernetes.io/projected/b0396d27-4305-4393-8d0e-f9a5cff12337-kube-api-access-qmhw2\") pod \"network-check-target-xppkv\" (UID: \"b0396d27-4305-4393-8d0e-f9a5cff12337\") " pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:47:27.370402 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.370372 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mvlkh\"" Jul 14 19:47:27.373171 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.373156 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6w8gx" Jul 14 19:47:27.378284 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.378261 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bh66q" Jul 14 19:47:27.467193 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.467166 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xm9xl\"" Jul 14 19:47:27.475383 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.475363 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:47:27.513477 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.513449 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6w8gx"] Jul 14 19:47:27.514908 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:47:27.514885 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f090a3_021a_4185_bcc8_7c00d7d56b77.slice/crio-9bb2d5a7cad0bb965f29b092c4f830160dd8b580acf37eb8c73512c1496a01bd WatchSource:0}: Error finding container 9bb2d5a7cad0bb965f29b092c4f830160dd8b580acf37eb8c73512c1496a01bd: Status 404 returned error can't find the container with id 9bb2d5a7cad0bb965f29b092c4f830160dd8b580acf37eb8c73512c1496a01bd Jul 14 19:47:27.533682 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.533650 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bh66q"] Jul 14 19:47:27.536005 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:47:27.535976 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c6f5f5_7f01_421d_bb94_170b7a1cda67.slice/crio-a83d953d4dcd8252f676e50200a45eefc6901e7bd9c86447f428dafe570011ed WatchSource:0}: Error finding container a83d953d4dcd8252f676e50200a45eefc6901e7bd9c86447f428dafe570011ed: Status 404 returned error can't find the container with id a83d953d4dcd8252f676e50200a45eefc6901e7bd9c86447f428dafe570011ed Jul 14 19:47:27.605698 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.605668 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xppkv"] Jul 14 19:47:27.608502 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:47:27.608477 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0396d27_4305_4393_8d0e_f9a5cff12337.slice/crio-a2b0d2aa5c5927ad57762ff734b5e2447bbf99f81e68ced17bb8fe4bc4b34b97 WatchSource:0}: Error finding container a2b0d2aa5c5927ad57762ff734b5e2447bbf99f81e68ced17bb8fe4bc4b34b97: Status 404 returned error can't find the container with id a2b0d2aa5c5927ad57762ff734b5e2447bbf99f81e68ced17bb8fe4bc4b34b97 Jul 14 19:47:27.618709 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.618684 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xppkv" event={"ID":"b0396d27-4305-4393-8d0e-f9a5cff12337","Type":"ContainerStarted","Data":"a2b0d2aa5c5927ad57762ff734b5e2447bbf99f81e68ced17bb8fe4bc4b34b97"} Jul 14 19:47:27.619714 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.619692 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bh66q" event={"ID":"a2c6f5f5-7f01-421d-bb94-170b7a1cda67","Type":"ContainerStarted","Data":"a83d953d4dcd8252f676e50200a45eefc6901e7bd9c86447f428dafe570011ed"} Jul 14 19:47:27.620625 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:27.620603 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6w8gx" event={"ID":"d3f090a3-021a-4185-bcc8-7c00d7d56b77","Type":"ContainerStarted","Data":"9bb2d5a7cad0bb965f29b092c4f830160dd8b580acf37eb8c73512c1496a01bd"} Jul 14 19:47:29.594662 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:29.594627 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9fb96" Jul 14 19:47:32.367959 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:32.367914 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:32.368422 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:32.367993 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:32.369669 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:32.369629 2557 patch_prober.go:28] interesting pod/console-5888c58fc-qzxlt container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.10:8443/health\": dial tcp 10.133.0.10:8443: connect: connection refused" start-of-body= Jul 14 19:47:32.369795 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:32.369681 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5888c58fc-qzxlt" podUID="d7742d52-e657-447d-a209-61d317713990" containerName="console" probeResult="failure" output="Get \"https://10.133.0.10:8443/health\": dial tcp 10.133.0.10:8443: connect: connection refused" Jul 14 19:47:32.652740 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:32.652700 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xppkv" event={"ID":"b0396d27-4305-4393-8d0e-f9a5cff12337","Type":"ContainerStarted","Data":"95962dd873d9510eb3f916f005706ecc3d1d1eed84015bfa94151adaa6215152"} Jul 14 19:47:32.652918 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:32.652800 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:47:32.654275 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:32.654241 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bh66q" event={"ID":"a2c6f5f5-7f01-421d-bb94-170b7a1cda67","Type":"ContainerStarted","Data":"8a0643bfcee75c4ea533f828de9040243826039e8bc43b78e9bf70beff9d6200"} Jul 14 19:47:32.655699 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:32.655671 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6w8gx" event={"ID":"d3f090a3-021a-4185-bcc8-7c00d7d56b77","Type":"ContainerStarted","Data":"5ea65423fe748f8f5a2e1e03cd600de12e36d313f5d6df05e1affba303732fb5"} Jul 14 19:47:32.687099 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:32.686958 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6w8gx" podStartSLOduration=65.680955035 podStartE2EDuration="1m10.68693932s" totalImagesPullingTime="5.005984285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:32.686938309 +0000 UTC m=+70.766684122" watchObservedRunningTime="2026-07-14 19:47:32.68693932 +0000 UTC m=+70.766685109" Jul 14 19:47:32.687314 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:32.687282 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xppkv" podStartSLOduration=65.777455545 podStartE2EDuration="1m10.687272988s" totalImagesPullingTime="4.909817443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:32.670147078 +0000 UTC m=+70.749892871" watchObservedRunningTime="2026-07-14 19:47:32.687272988 +0000 UTC m=+70.767018784" Jul 14 19:47:33.660824 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:33.660780 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bh66q" event={"ID":"a2c6f5f5-7f01-421d-bb94-170b7a1cda67","Type":"ContainerStarted","Data":"1ea43a026d80bd7a232f2a4f9ff2b95a0e994476e887536ef13e7d165ab2c46c"} Jul 14 19:47:33.686576 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:33.686532 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bh66q" podStartSLOduration=66.723124993 podStartE2EDuration="1m11.686518345s" totalImagesPullingTime="4.963393352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:33.686406849 +0000 UTC m=+71.766152641" watchObservedRunningTime="2026-07-14 19:47:33.686518345 +0000 UTC m=+71.766264136" Jul 14 19:47:36.358783 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.358758 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:36.362405 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.362376 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:47:36.374559 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.374534 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-zzxjn"] Jul 14 19:47:36.377450 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.377435 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:36.379814 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.379786 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bc9nz\"" Jul 14 19:47:36.379988 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.379970 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 19:47:36.380085 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.380034 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 14 19:47:36.380743 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.380723 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 19:47:36.380823 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.380772 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 19:47:36.380823 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.380782 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 14 19:47:36.388872 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.388851 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-zzxjn"] Jul 14 19:47:36.410017 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.409982 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctzd7\" (UniqueName: \"kubernetes.io/projected/0d559014-bde8-4125-b1d5-7ae7f113c52b-kube-api-access-ctzd7\") pod \"prometheus-operator-64785b6675-zzxjn\" (UID: \"0d559014-bde8-4125-b1d5-7ae7f113c52b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:36.410184 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.410090 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d559014-bde8-4125-b1d5-7ae7f113c52b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-zzxjn\" (UID: \"0d559014-bde8-4125-b1d5-7ae7f113c52b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:36.410184 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.410123 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d559014-bde8-4125-b1d5-7ae7f113c52b-metrics-client-ca\") pod \"prometheus-operator-64785b6675-zzxjn\" (UID: \"0d559014-bde8-4125-b1d5-7ae7f113c52b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:36.410311 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.410255 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d559014-bde8-4125-b1d5-7ae7f113c52b-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-zzxjn\" (UID: \"0d559014-bde8-4125-b1d5-7ae7f113c52b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:36.510905 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.510865 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d559014-bde8-4125-b1d5-7ae7f113c52b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-zzxjn\" (UID: \"0d559014-bde8-4125-b1d5-7ae7f113c52b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:36.510905 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.510909 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d559014-bde8-4125-b1d5-7ae7f113c52b-metrics-client-ca\") pod \"prometheus-operator-64785b6675-zzxjn\" (UID: \"0d559014-bde8-4125-b1d5-7ae7f113c52b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:36.511183 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.510930 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d559014-bde8-4125-b1d5-7ae7f113c52b-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-zzxjn\" (UID: \"0d559014-bde8-4125-b1d5-7ae7f113c52b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:36.511183 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.510952 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzd7\" (UniqueName: \"kubernetes.io/projected/0d559014-bde8-4125-b1d5-7ae7f113c52b-kube-api-access-ctzd7\") pod \"prometheus-operator-64785b6675-zzxjn\" (UID: \"0d559014-bde8-4125-b1d5-7ae7f113c52b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:36.511183 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:36.511078 2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 14 19:47:36.511183 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:36.511145 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d559014-bde8-4125-b1d5-7ae7f113c52b-prometheus-operator-tls podName:0d559014-bde8-4125-b1d5-7ae7f113c52b nodeName:}" failed. No retries permitted until 2026-07-14 19:47:37.011124139 +0000 UTC m=+75.090869926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/0d559014-bde8-4125-b1d5-7ae7f113c52b-prometheus-operator-tls") pod "prometheus-operator-64785b6675-zzxjn" (UID: "0d559014-bde8-4125-b1d5-7ae7f113c52b") : secret "prometheus-operator-tls" not found Jul 14 19:47:36.511599 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.511574 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d559014-bde8-4125-b1d5-7ae7f113c52b-metrics-client-ca\") pod \"prometheus-operator-64785b6675-zzxjn\" (UID: \"0d559014-bde8-4125-b1d5-7ae7f113c52b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:36.513272 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.513247 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d559014-bde8-4125-b1d5-7ae7f113c52b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-zzxjn\" (UID: \"0d559014-bde8-4125-b1d5-7ae7f113c52b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:36.519428 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:36.519400 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctzd7\" (UniqueName: \"kubernetes.io/projected/0d559014-bde8-4125-b1d5-7ae7f113c52b-kube-api-access-ctzd7\") pod \"prometheus-operator-64785b6675-zzxjn\" (UID: \"0d559014-bde8-4125-b1d5-7ae7f113c52b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:37.013696 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:37.013649 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d559014-bde8-4125-b1d5-7ae7f113c52b-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-zzxjn\" (UID: \"0d559014-bde8-4125-b1d5-7ae7f113c52b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:37.013852 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:37.013792 2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 14 19:47:37.013906 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:37.013855 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d559014-bde8-4125-b1d5-7ae7f113c52b-prometheus-operator-tls podName:0d559014-bde8-4125-b1d5-7ae7f113c52b nodeName:}" failed. No retries permitted until 2026-07-14 19:47:38.013841117 +0000 UTC m=+76.093586887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/0d559014-bde8-4125-b1d5-7ae7f113c52b-prometheus-operator-tls") pod "prometheus-operator-64785b6675-zzxjn" (UID: "0d559014-bde8-4125-b1d5-7ae7f113c52b") : secret "prometheus-operator-tls" not found Jul 14 19:47:38.020748 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:38.020709 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d559014-bde8-4125-b1d5-7ae7f113c52b-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-zzxjn\" (UID: \"0d559014-bde8-4125-b1d5-7ae7f113c52b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:38.023050 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:38.023029 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0d559014-bde8-4125-b1d5-7ae7f113c52b-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-zzxjn\" (UID: \"0d559014-bde8-4125-b1d5-7ae7f113c52b\") " pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:38.187373 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:38.187338 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" Jul 14 19:47:38.315442 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:38.315374 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-zzxjn"] Jul 14 19:47:38.318398 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:47:38.318374 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d559014_bde8_4125_b1d5_7ae7f113c52b.slice/crio-cf9c4476f998b068f5a4696373dfbcc929b89aa851489ee32f53602d2a0ceb46 WatchSource:0}: Error finding container cf9c4476f998b068f5a4696373dfbcc929b89aa851489ee32f53602d2a0ceb46: Status 404 returned error can't find the container with id cf9c4476f998b068f5a4696373dfbcc929b89aa851489ee32f53602d2a0ceb46 Jul 14 19:47:38.678759 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:38.678722 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" event={"ID":"0d559014-bde8-4125-b1d5-7ae7f113c52b","Type":"ContainerStarted","Data":"cf9c4476f998b068f5a4696373dfbcc929b89aa851489ee32f53602d2a0ceb46"} Jul 14 19:47:40.688262 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:40.688225 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" event={"ID":"0d559014-bde8-4125-b1d5-7ae7f113c52b","Type":"ContainerStarted","Data":"6a54aa9191d36e53d71baaa64b90a7e7866abbb4c80d0da6cc163a4117224478"} Jul 14 19:47:40.688645 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:40.688282 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" event={"ID":"0d559014-bde8-4125-b1d5-7ae7f113c52b","Type":"ContainerStarted","Data":"a49c88ab8fbc1bb89b8848bfbd891a3d15f14b456bfdfb6e891ced10dccfecd4"} Jul 14 19:47:40.713327 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:40.713250 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-zzxjn" podStartSLOduration=3.002140272 podStartE2EDuration="4.713233901s" totalImagesPullingTime="1.711093629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:40.711803462 +0000 UTC m=+78.791549254" watchObservedRunningTime="2026-07-14 19:47:40.713233901 +0000 UTC m=+78.792979715" Jul 14 19:47:42.371670 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.371640 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:42.375555 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.375530 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:47:42.451391 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.451363 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59fb8745d5-wb5dk"] Jul 14 19:47:42.810537 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.810450 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-cqtfv"] Jul 14 19:47:42.813726 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.813705 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.815776 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.815753 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-brcrp\"" Jul 14 19:47:42.816540 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.816509 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 14 19:47:42.816540 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.816514 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 14 19:47:42.816696 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.816593 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 14 19:47:42.823329 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.823306 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c4sth"] Jul 14 19:47:42.826218 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.826199 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:42.826841 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.826821 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-cqtfv"] Jul 14 19:47:42.829495 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.829474 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 19:47:42.830139 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.830118 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 19:47:42.830253 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.830238 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6x7x8\"" Jul 14 19:47:42.830527 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.830503 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 19:47:42.853963 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.853932 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0b73d9ec-522c-4b17-ab16-b21fdae87b75-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.854111 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.853969 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b73d9ec-522c-4b17-ab16-b21fdae87b75-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.854111 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.853991 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.854111 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.854046 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.854215 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.854142 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4hpm\" (UniqueName: \"kubernetes.io/projected/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-api-access-q4hpm\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.854215 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.854184 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.955531 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.955497 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-node-exporter-wtmp\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:42.955531 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.955532 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-node-exporter-textfile\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:42.955753 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.955561 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0b73d9ec-522c-4b17-ab16-b21fdae87b75-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.955753 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.955606 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqn4b\" (UniqueName: \"kubernetes.io/projected/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-kube-api-access-wqn4b\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:42.955753 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.955642 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-root\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:42.955753 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.955658 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:42.955753 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.955682 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-sys\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:42.955753 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.955700 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b73d9ec-522c-4b17-ab16-b21fdae87b75-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.955753 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.955717 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.956089 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.955775 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.956089 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.955804 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-node-exporter-tls\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:42.956089 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.955844 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4hpm\" (UniqueName: \"kubernetes.io/projected/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-api-access-q4hpm\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.956089 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.955864 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-metrics-client-ca\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:42.956089 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.955898 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:42.956089 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:42.955903 2557 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 14 19:47:42.956089 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.955944 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.956089 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:42.955981 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-state-metrics-tls podName:0b73d9ec-522c-4b17-ab16-b21fdae87b75 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:43.455958712 +0000 UTC m=+81.535704493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-cqtfv" (UID: "0b73d9ec-522c-4b17-ab16-b21fdae87b75") : secret "kube-state-metrics-tls" not found Jul 14 19:47:42.956089 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.956029 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0b73d9ec-522c-4b17-ab16-b21fdae87b75-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.956483 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.956462 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.956546 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.956466 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b73d9ec-522c-4b17-ab16-b21fdae87b75-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.958320 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.958301 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:42.965174 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:42.965150 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4hpm\" (UniqueName: \"kubernetes.io/projected/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-api-access-q4hpm\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:43.056425 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.056390 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqn4b\" (UniqueName: \"kubernetes.io/projected/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-kube-api-access-wqn4b\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.056425 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.056428 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-root\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.056642 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.056445 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.056642 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.056468 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-sys\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.056642 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.056514 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-root\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.056642 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.056526 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-sys\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.056642 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.056594 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-node-exporter-tls\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.056642 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.056638 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-metrics-client-ca\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.056894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.056675 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.056894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.056710 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-node-exporter-wtmp\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.056894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.056738 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-node-exporter-textfile\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.056894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.056851 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-node-exporter-wtmp\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.057102 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.057089 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-node-exporter-textfile\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.057102 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.057096 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.057342 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.057321 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-metrics-client-ca\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.058975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.058950 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.059085 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.059049 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-node-exporter-tls\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.070567 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.070511 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqn4b\" (UniqueName: \"kubernetes.io/projected/222df6f4-91ed-4e2f-842d-ddf0d767a6d8-kube-api-access-wqn4b\") pod \"node-exporter-c4sth\" (UID: \"222df6f4-91ed-4e2f-842d-ddf0d767a6d8\") " pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.134889 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.134859 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c4sth" Jul 14 19:47:43.146477 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:47:43.146450 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222df6f4_91ed_4e2f_842d_ddf0d767a6d8.slice/crio-e465e44c98b54fd6f5c9f527dee636ff1a1f427599945388693d16f082bf142e WatchSource:0}: Error finding container e465e44c98b54fd6f5c9f527dee636ff1a1f427599945388693d16f082bf142e: Status 404 returned error can't find the container with id e465e44c98b54fd6f5c9f527dee636ff1a1f427599945388693d16f082bf142e Jul 14 19:47:43.459277 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.459242 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:43.459656 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:43.459357 2557 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 14 19:47:43.459656 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:47:43.459408 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-state-metrics-tls podName:0b73d9ec-522c-4b17-ab16-b21fdae87b75 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:44.459394568 +0000 UTC m=+82.539140338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-cqtfv" (UID: "0b73d9ec-522c-4b17-ab16-b21fdae87b75") : secret "kube-state-metrics-tls" not found Jul 14 19:47:43.700540 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:43.700504 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4sth" event={"ID":"222df6f4-91ed-4e2f-842d-ddf0d767a6d8","Type":"ContainerStarted","Data":"e465e44c98b54fd6f5c9f527dee636ff1a1f427599945388693d16f082bf142e"} Jul 14 19:47:44.468398 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:44.468281 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:44.470621 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:44.470597 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b73d9ec-522c-4b17-ab16-b21fdae87b75-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-cqtfv\" (UID: \"0b73d9ec-522c-4b17-ab16-b21fdae87b75\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:44.623148 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:44.623110 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" Jul 14 19:47:44.707980 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:44.707952 2557 generic.go:356] "Generic (PLEG): container finished" podID="222df6f4-91ed-4e2f-842d-ddf0d767a6d8" containerID="ccda26052a1574ba268bab668b38f2a89f549bc8f16294125c2fcaf17966cd0d" exitCode=0 Jul 14 19:47:44.708123 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:44.708047 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4sth" event={"ID":"222df6f4-91ed-4e2f-842d-ddf0d767a6d8","Type":"ContainerDied","Data":"ccda26052a1574ba268bab668b38f2a89f549bc8f16294125c2fcaf17966cd0d"} Jul 14 19:47:44.757975 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:44.757946 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-cqtfv"] Jul 14 19:47:44.761037 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:47:44.761011 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b73d9ec_522c_4b17_ab16_b21fdae87b75.slice/crio-2b1f54ad2a1eae0414cee8364bac59e6ec74bbee75a5f42fcfbeefa269b0d3f6 WatchSource:0}: Error finding container 2b1f54ad2a1eae0414cee8364bac59e6ec74bbee75a5f42fcfbeefa269b0d3f6: Status 404 returned error can't find the container with id 2b1f54ad2a1eae0414cee8364bac59e6ec74bbee75a5f42fcfbeefa269b0d3f6 Jul 14 19:47:45.714965 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.714924 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4sth" event={"ID":"222df6f4-91ed-4e2f-842d-ddf0d767a6d8","Type":"ContainerStarted","Data":"b4488168a2747352d3cf30b8a9598ffa91cec10a35b80ab6e084742c501e2fbf"} Jul 14 19:47:45.715395 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.714993 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c4sth" event={"ID":"222df6f4-91ed-4e2f-842d-ddf0d767a6d8","Type":"ContainerStarted","Data":"3b357eb4dd0de3e58d0f2f4ddadbe06ecc3f4e2a9aff5060e82768c0e678a776"} Jul 14 19:47:45.716148 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.716123 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" event={"ID":"0b73d9ec-522c-4b17-ab16-b21fdae87b75","Type":"ContainerStarted","Data":"2b1f54ad2a1eae0414cee8364bac59e6ec74bbee75a5f42fcfbeefa269b0d3f6"} Jul 14 19:47:45.870171 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.870113 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c4sth" podStartSLOduration=2.447776045 podStartE2EDuration="3.870094965s" totalImagesPullingTime="1.353511083s" totalInitContainerRuntime="68.807837ms" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:45.735926409 +0000 UTC m=+83.815672223" watchObservedRunningTime="2026-07-14 19:47:45.870094965 +0000 UTC m=+83.949840756" Jul 14 19:47:45.871119 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.871097 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-65889bfdcf-qzcqn"] Jul 14 19:47:45.874619 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.874597 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:45.876920 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.876893 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 14 19:47:45.876920 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.876916 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 14 19:47:45.877134 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.876893 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 14 19:47:45.877134 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.876895 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8krr52hc9e0j9\"" Jul 14 19:47:45.877134 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.877089 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-j9d6b\"" Jul 14 19:47:45.877258 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.877208 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 14 19:47:45.877533 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.877512 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 14 19:47:45.897784 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.897760 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-65889bfdcf-qzcqn"] Jul 14 19:47:45.980344 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.980263 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s76jg\" (UniqueName: \"kubernetes.io/projected/2aca72f4-54b5-4676-96eb-78ebd8f25877-kube-api-access-s76jg\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:45.980344 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.980295 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-grpc-tls\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:45.980344 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.980312 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aca72f4-54b5-4676-96eb-78ebd8f25877-metrics-client-ca\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:45.980344 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.980333 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:45.980597 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.980363 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:45.980597 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.980442 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-thanos-querier-tls\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:45.980597 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.980501 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:45.980597 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:45.980521 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.081278 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.081239 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s76jg\" (UniqueName: \"kubernetes.io/projected/2aca72f4-54b5-4676-96eb-78ebd8f25877-kube-api-access-s76jg\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.081278 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.081280 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-grpc-tls\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.081569 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.081298 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aca72f4-54b5-4676-96eb-78ebd8f25877-metrics-client-ca\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.081569 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.081315 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.081569 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.081334 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.081569 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.081360 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-thanos-querier-tls\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.081569 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.081403 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.081569 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.081419 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.082277 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.082216 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aca72f4-54b5-4676-96eb-78ebd8f25877-metrics-client-ca\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.084015 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.083983 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.084150 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.084127 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-grpc-tls\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.084359 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.084339 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.084569 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.084554 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-thanos-querier-tls\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.084683 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.084660 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.084715 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.084670 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2aca72f4-54b5-4676-96eb-78ebd8f25877-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.091110 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.091089 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s76jg\" (UniqueName: \"kubernetes.io/projected/2aca72f4-54b5-4676-96eb-78ebd8f25877-kube-api-access-s76jg\") pod \"thanos-querier-65889bfdcf-qzcqn\" (UID: \"2aca72f4-54b5-4676-96eb-78ebd8f25877\") " pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.184272 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.184233 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:46.332680 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.332647 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-65889bfdcf-qzcqn"] Jul 14 19:47:46.335941 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:47:46.335916 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aca72f4_54b5_4676_96eb_78ebd8f25877.slice/crio-cc8ed2d69af2c6eeadfc704640207886b905f4c989d40225d9a86cabf66c3b22 WatchSource:0}: Error finding container cc8ed2d69af2c6eeadfc704640207886b905f4c989d40225d9a86cabf66c3b22: Status 404 returned error can't find the container with id cc8ed2d69af2c6eeadfc704640207886b905f4c989d40225d9a86cabf66c3b22 Jul 14 19:47:46.721618 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:46.721583 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" event={"ID":"2aca72f4-54b5-4676-96eb-78ebd8f25877","Type":"ContainerStarted","Data":"cc8ed2d69af2c6eeadfc704640207886b905f4c989d40225d9a86cabf66c3b22"} Jul 14 19:47:48.731763 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:48.731736 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" event={"ID":"0b73d9ec-522c-4b17-ab16-b21fdae87b75","Type":"ContainerStarted","Data":"bcbe27b848533a3ce17ce56554088f3de97cb5c4be114bab76036a5d0d1eb52c"} Jul 14 19:47:48.732081 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:48.731796 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" event={"ID":"0b73d9ec-522c-4b17-ab16-b21fdae87b75","Type":"ContainerStarted","Data":"3772118e43a94d59abde6163a3a4faed795ccdc3e9570554ff30c4f34aa0b6bc"} Jul 14 19:47:49.033497 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.033458 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 19:47:49.047274 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.047245 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.049965 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.049823 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 14 19:47:49.049965 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.049868 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 14 19:47:49.050172 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.050144 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gbsn2\"" Jul 14 19:47:49.050229 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.050203 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 14 19:47:49.050229 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.050212 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 14 19:47:49.050453 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.050433 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 14 19:47:49.050531 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.050434 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 14 19:47:49.050531 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.050496 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 14 19:47:49.051406 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.050797 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 14 19:47:49.051406 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.050832 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 14 19:47:49.051406 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.050798 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 14 19:47:49.051406 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.050804 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 14 19:47:49.051406 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.051269 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 14 19:47:49.051730 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.051576 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-h343e48agr0e\"" Jul 14 19:47:49.051730 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.051720 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 14 19:47:49.054087 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.053784 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 14 19:47:49.057368 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.057281 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 19:47:49.058165 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.057944 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 14 19:47:49.108384 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108334 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec172125-9bd4-4684-9a94-34e9e4f4af50-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.108384 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108376 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.108587 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108407 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.108587 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108434 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec172125-9bd4-4684-9a94-34e9e4f4af50-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.108587 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108457 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9s6c\" (UniqueName: \"kubernetes.io/projected/ec172125-9bd4-4684-9a94-34e9e4f4af50-kube-api-access-s9s6c\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.108587 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108535 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.108587 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108572 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.108844 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108602 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-web-config\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.108844 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108635 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-config\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.108844 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108656 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.108844 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108712 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.108844 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108773 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.108844 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108821 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.109153 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108847 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec172125-9bd4-4684-9a94-34e9e4f4af50-config-out\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.109153 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108869 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.109153 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108933 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.109153 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108959 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.109153 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108983 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.109153 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.108997 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.109153 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.109013 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.209884 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.209857 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.211187 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.210956 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.211187 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211009 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.211187 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.210638 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.211187 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211036 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.211187 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211124 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.211187 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211171 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec172125-9bd4-4684-9a94-34e9e4f4af50-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.211512 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211210 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.211512 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211236 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.211728 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211703 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec172125-9bd4-4684-9a94-34e9e4f4af50-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.211802 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211745 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9s6c\" (UniqueName: \"kubernetes.io/projected/ec172125-9bd4-4684-9a94-34e9e4f4af50-kube-api-access-s9s6c\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.211802 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211776 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.211895 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211801 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.211895 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211828 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-web-config\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.211895 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211875 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-config\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.212039 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211909 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.212039 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211944 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.212039 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211971 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.212039 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.211998 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.212039 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.212029 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec172125-9bd4-4684-9a94-34e9e4f4af50-config-out\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.212298 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.212070 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.220894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.215841 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.220894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.216263 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.220894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.217007 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.220894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.217391 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ec172125-9bd4-4684-9a94-34e9e4f4af50-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.220894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.217410 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.220894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.217518 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.220894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.218186 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ec172125-9bd4-4684-9a94-34e9e4f4af50-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.220894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.218357 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.220894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.219714 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-config\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.220894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.219864 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.220894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.220014 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.220894 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.220481 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.221531 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.220960 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.221584 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.221563 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ec172125-9bd4-4684-9a94-34e9e4f4af50-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.225217 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.225168 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.225515 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.225381 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-web-config\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.226451 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.226433 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ec172125-9bd4-4684-9a94-34e9e4f4af50-config-out\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.227276 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.227250 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ec172125-9bd4-4684-9a94-34e9e4f4af50-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.228453 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.228429 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9s6c\" (UniqueName: \"kubernetes.io/projected/ec172125-9bd4-4684-9a94-34e9e4f4af50-kube-api-access-s9s6c\") pod \"prometheus-k8s-0\" (UID: \"ec172125-9bd4-4684-9a94-34e9e4f4af50\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.362469 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.362430 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:47:49.523941 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.523911 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 19:47:49.527166 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:47:49.527137 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec172125_9bd4_4684_9a94_34e9e4f4af50.slice/crio-5df824fdac070d6458af5fa189de6b92d2ce93e096aff8384567908d818fe122 WatchSource:0}: Error finding container 5df824fdac070d6458af5fa189de6b92d2ce93e096aff8384567908d818fe122: Status 404 returned error can't find the container with id 5df824fdac070d6458af5fa189de6b92d2ce93e096aff8384567908d818fe122 Jul 14 19:47:49.738507 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.738419 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" event={"ID":"2aca72f4-54b5-4676-96eb-78ebd8f25877","Type":"ContainerStarted","Data":"114b81f29055c4054b5f5045e583cf5c1317a73d801391624ef076e8fd9bafd3"} Jul 14 19:47:49.738507 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.738488 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" event={"ID":"2aca72f4-54b5-4676-96eb-78ebd8f25877","Type":"ContainerStarted","Data":"4162db8e9684c0d91878b194afdd5db647c7888d3ff9773f863be6bb43c741ee"} Jul 14 19:47:49.738911 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.738524 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" event={"ID":"2aca72f4-54b5-4676-96eb-78ebd8f25877","Type":"ContainerStarted","Data":"a6e2ccabd18e72246ec14ffdf3903d4f7185dc939700255f761cc20543ec6e0b"} Jul 14 19:47:49.739681 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.739650 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec172125-9bd4-4684-9a94-34e9e4f4af50","Type":"ContainerStarted","Data":"5df824fdac070d6458af5fa189de6b92d2ce93e096aff8384567908d818fe122"} Jul 14 19:47:49.741437 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.741415 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" event={"ID":"0b73d9ec-522c-4b17-ab16-b21fdae87b75","Type":"ContainerStarted","Data":"b28aff33a8d64dcb48ba5fecb5f419beeeeebfd1c435346f2e08623d552479c7"} Jul 14 19:47:49.760634 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:49.760589 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-cqtfv" podStartSLOduration=4.700211049 podStartE2EDuration="7.76057462s" totalImagesPullingTime="3.060363571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:49.759907589 +0000 UTC m=+87.839653381" watchObservedRunningTime="2026-07-14 19:47:49.76057462 +0000 UTC m=+87.840320412" Jul 14 19:47:51.756741 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:51.756700 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" event={"ID":"2aca72f4-54b5-4676-96eb-78ebd8f25877","Type":"ContainerStarted","Data":"7f79436bf9022b399330a4197aa259aa0a2b6b2d4a40c62ab24b869570f27f6d"} Jul 14 19:47:51.757138 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:51.756775 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" event={"ID":"2aca72f4-54b5-4676-96eb-78ebd8f25877","Type":"ContainerStarted","Data":"4c521f0bc39709891f0961a06460b8cf3a4abdf16213287513393fa452a2204f"} Jul 14 19:47:51.757138 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:51.756806 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" event={"ID":"2aca72f4-54b5-4676-96eb-78ebd8f25877","Type":"ContainerStarted","Data":"b0b532ca55e35a4b547d890ec6362f791eeed3436295dcc31cc4b38ba4afa1a0"} Jul 14 19:47:51.757138 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:51.756851 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:51.781078 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:51.781006 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" podStartSLOduration=2.40865886 podStartE2EDuration="6.78098755s" totalImagesPullingTime="4.37232869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:51.779113032 +0000 UTC m=+89.858858853" watchObservedRunningTime="2026-07-14 19:47:51.78098755 +0000 UTC m=+89.860733341" Jul 14 19:47:52.765341 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:52.765258 2557 generic.go:356] "Generic (PLEG): container finished" podID="ec172125-9bd4-4684-9a94-34e9e4f4af50" containerID="e602b0f58af673ea060b431e91fa28cf3e0593921cc3f4477ca0af52522024f1" exitCode=0 Jul 14 19:47:52.765696 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:52.765345 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec172125-9bd4-4684-9a94-34e9e4f4af50","Type":"ContainerDied","Data":"e602b0f58af673ea060b431e91fa28cf3e0593921cc3f4477ca0af52522024f1"} Jul 14 19:47:54.406530 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.406492 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6894598bd5-p94qv"] Jul 14 19:47:54.459782 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.459743 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.459958 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.459631 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6894598bd5-p94qv"] Jul 14 19:47:54.557539 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.557498 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e649768-f5f6-4226-93ef-4123627698a1-console-oauth-config\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.557539 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.557543 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-console-config\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.557782 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.557576 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlr65\" (UniqueName: \"kubernetes.io/projected/8e649768-f5f6-4226-93ef-4123627698a1-kube-api-access-jlr65\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.557782 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.557679 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-service-ca\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.557782 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.557765 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-oauth-serving-cert\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.557931 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.557801 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e649768-f5f6-4226-93ef-4123627698a1-console-serving-cert\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.557931 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.557851 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-trusted-ca-bundle\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.659238 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.659151 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-oauth-serving-cert\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.659238 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.659195 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e649768-f5f6-4226-93ef-4123627698a1-console-serving-cert\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.659238 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.659232 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-trusted-ca-bundle\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.659514 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.659268 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e649768-f5f6-4226-93ef-4123627698a1-console-oauth-config\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.659514 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.659288 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-console-config\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.659514 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.659360 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlr65\" (UniqueName: \"kubernetes.io/projected/8e649768-f5f6-4226-93ef-4123627698a1-kube-api-access-jlr65\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.659714 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.659690 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-service-ca\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.660082 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.660013 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-console-config\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.660082 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.660025 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-oauth-serving-cert\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.660293 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.660271 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-trusted-ca-bundle\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.660350 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.660298 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-service-ca\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.661915 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.661892 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e649768-f5f6-4226-93ef-4123627698a1-console-serving-cert\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.662043 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.662021 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e649768-f5f6-4226-93ef-4123627698a1-console-oauth-config\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.668391 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.668367 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlr65\" (UniqueName: \"kubernetes.io/projected/8e649768-f5f6-4226-93ef-4123627698a1-kube-api-access-jlr65\") pod \"console-6894598bd5-p94qv\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.769704 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.769669 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:47:54.917754 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:54.917729 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6894598bd5-p94qv"] Jul 14 19:47:54.920212 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:47:54.920176 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e649768_f5f6_4226_93ef_4123627698a1.slice/crio-fcd396725ca7ac1818d6510e0cbabf3dffda28abc893faa97ed11041155aa8b9 WatchSource:0}: Error finding container fcd396725ca7ac1818d6510e0cbabf3dffda28abc893faa97ed11041155aa8b9: Status 404 returned error can't find the container with id fcd396725ca7ac1818d6510e0cbabf3dffda28abc893faa97ed11041155aa8b9 Jul 14 19:47:55.781320 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:55.781281 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6894598bd5-p94qv" event={"ID":"8e649768-f5f6-4226-93ef-4123627698a1","Type":"ContainerStarted","Data":"515cdc2c5bebecb480848974ee79d5b9bd352c27bc347e471adf7b7d07cc7a32"} Jul 14 19:47:55.781790 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:55.781340 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6894598bd5-p94qv" event={"ID":"8e649768-f5f6-4226-93ef-4123627698a1","Type":"ContainerStarted","Data":"fcd396725ca7ac1818d6510e0cbabf3dffda28abc893faa97ed11041155aa8b9"} Jul 14 19:47:55.803918 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:55.803869 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6894598bd5-p94qv" podStartSLOduration=1.803853653 podStartE2EDuration="1.803853653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:55.80273212 +0000 UTC m=+93.882477913" watchObservedRunningTime="2026-07-14 19:47:55.803853653 +0000 UTC m=+93.883599446" Jul 14 19:47:57.772703 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:57.772626 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-65889bfdcf-qzcqn" Jul 14 19:47:57.792444 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:57.792412 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec172125-9bd4-4684-9a94-34e9e4f4af50","Type":"ContainerStarted","Data":"4b85e241366970a1a6459262f0c1692cccba0f6dbac316c3205da33deed7d2d3"} Jul 14 19:47:57.792610 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:57.792497 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec172125-9bd4-4684-9a94-34e9e4f4af50","Type":"ContainerStarted","Data":"3241e76def1063f390444ea23a5d444bc5517077b1d23fa344efaf2fc9b60ce1"} Jul 14 19:47:57.792610 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:57.792529 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec172125-9bd4-4684-9a94-34e9e4f4af50","Type":"ContainerStarted","Data":"0882301daf33358e2f90865ced7db5044d6886917424a1eaeb67718ef4a8c3a2"} Jul 14 19:47:57.792610 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:57.792553 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec172125-9bd4-4684-9a94-34e9e4f4af50","Type":"ContainerStarted","Data":"691c114cc1af9a265c830919aa65464b43d4f390c64af59544f9ab0e16345e5c"} Jul 14 19:47:57.792610 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:57.792576 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec172125-9bd4-4684-9a94-34e9e4f4af50","Type":"ContainerStarted","Data":"cf6c1e5d2bbf6690e814cdf17cc6ca136566a6548e0e9586593ac813ae943e6f"} Jul 14 19:47:57.792610 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:57.792599 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ec172125-9bd4-4684-9a94-34e9e4f4af50","Type":"ContainerStarted","Data":"f9124cbd3a6402e8c5011177df16d9dc90238ff63fbc530f5186292eb6af2f41"} Jul 14 19:47:57.826110 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:57.825630 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.585961568 podStartE2EDuration="8.825609257s" totalImagesPullingTime="7.21418637s" totalInitContainerRuntime="25.461319ms" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:57.82350126 +0000 UTC m=+95.903247052" watchObservedRunningTime="2026-07-14 19:47:57.825609257 +0000 UTC m=+95.905355054" Jul 14 19:47:59.363322 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:47:59.363276 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:48:03.662810 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:03.662777 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xppkv" Jul 14 19:48:04.770033 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:04.769998 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:48:04.770532 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:04.770094 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:48:04.774780 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:04.774764 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:48:04.820140 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:04.820113 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:48:04.878339 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:04.878307 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5888c58fc-qzxlt"] Jul 14 19:48:07.473706 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.473663 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59fb8745d5-wb5dk" podUID="719cb6c1-7370-4e96-bd1a-6701eec6bb6a" containerName="console" containerID="cri-o://e16a195d12cee5c54c2f0c3fb125149064801eefc9c8ba3b6b838ac9ea555ab2" gracePeriod=15 Jul 14 19:48:07.737680 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.737614 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:48:07.829043 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.829008 2557 generic.go:356] "Generic (PLEG): container finished" podID="719cb6c1-7370-4e96-bd1a-6701eec6bb6a" containerID="e16a195d12cee5c54c2f0c3fb125149064801eefc9c8ba3b6b838ac9ea555ab2" exitCode=0 Jul 14 19:48:07.829243 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.829080 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59fb8745d5-wb5dk" Jul 14 19:48:07.829243 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.829103 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59fb8745d5-wb5dk" event={"ID":"719cb6c1-7370-4e96-bd1a-6701eec6bb6a","Type":"ContainerDied","Data":"e16a195d12cee5c54c2f0c3fb125149064801eefc9c8ba3b6b838ac9ea555ab2"} Jul 14 19:48:07.829243 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.829160 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59fb8745d5-wb5dk" event={"ID":"719cb6c1-7370-4e96-bd1a-6701eec6bb6a","Type":"ContainerDied","Data":"fecc3959dc3442532fa49c15a2d9d403f699e36db677078ddc6e015074f784e2"} Jul 14 19:48:07.829243 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.829181 2557 scope.go:117] "RemoveContainer" containerID="e16a195d12cee5c54c2f0c3fb125149064801eefc9c8ba3b6b838ac9ea555ab2" Jul 14 19:48:07.837122 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.837104 2557 scope.go:117] "RemoveContainer" containerID="e16a195d12cee5c54c2f0c3fb125149064801eefc9c8ba3b6b838ac9ea555ab2" Jul 14 19:48:07.837381 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:48:07.837357 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16a195d12cee5c54c2f0c3fb125149064801eefc9c8ba3b6b838ac9ea555ab2\": container with ID starting with e16a195d12cee5c54c2f0c3fb125149064801eefc9c8ba3b6b838ac9ea555ab2 not found: ID does not exist" containerID="e16a195d12cee5c54c2f0c3fb125149064801eefc9c8ba3b6b838ac9ea555ab2" Jul 14 19:48:07.837429 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.837389 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16a195d12cee5c54c2f0c3fb125149064801eefc9c8ba3b6b838ac9ea555ab2"} err="failed to get container status \"e16a195d12cee5c54c2f0c3fb125149064801eefc9c8ba3b6b838ac9ea555ab2\": rpc error: code = NotFound desc = could not find container \"e16a195d12cee5c54c2f0c3fb125149064801eefc9c8ba3b6b838ac9ea555ab2\": container with ID starting with e16a195d12cee5c54c2f0c3fb125149064801eefc9c8ba3b6b838ac9ea555ab2 not found: ID does not exist" Jul 14 19:48:07.883768 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.883737 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk96v\" (UniqueName: \"kubernetes.io/projected/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-kube-api-access-qk96v\") pod \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " Jul 14 19:48:07.883768 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.883772 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-oauth-config\") pod \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " Jul 14 19:48:07.883963 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.883808 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-service-ca\") pod \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " Jul 14 19:48:07.883963 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.883903 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-serving-cert\") pod \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " Jul 14 19:48:07.883963 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.883946 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-config\") pod \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " Jul 14 19:48:07.884139 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.883971 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-oauth-serving-cert\") pod \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\" (UID: \"719cb6c1-7370-4e96-bd1a-6701eec6bb6a\") " Jul 14 19:48:07.884343 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.884313 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-service-ca" (OuterVolumeSpecName: "service-ca") pod "719cb6c1-7370-4e96-bd1a-6701eec6bb6a" (UID: "719cb6c1-7370-4e96-bd1a-6701eec6bb6a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:48:07.884343 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.884325 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-config" (OuterVolumeSpecName: "console-config") pod "719cb6c1-7370-4e96-bd1a-6701eec6bb6a" (UID: "719cb6c1-7370-4e96-bd1a-6701eec6bb6a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:48:07.884486 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.884373 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "719cb6c1-7370-4e96-bd1a-6701eec6bb6a" (UID: "719cb6c1-7370-4e96-bd1a-6701eec6bb6a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:48:07.886232 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.886207 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "719cb6c1-7370-4e96-bd1a-6701eec6bb6a" (UID: "719cb6c1-7370-4e96-bd1a-6701eec6bb6a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:48:07.886338 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.886294 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "719cb6c1-7370-4e96-bd1a-6701eec6bb6a" (UID: "719cb6c1-7370-4e96-bd1a-6701eec6bb6a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:48:07.886391 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.886332 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-kube-api-access-qk96v" (OuterVolumeSpecName: "kube-api-access-qk96v") pod "719cb6c1-7370-4e96-bd1a-6701eec6bb6a" (UID: "719cb6c1-7370-4e96-bd1a-6701eec6bb6a"). InnerVolumeSpecName "kube-api-access-qk96v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:48:07.985386 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.985355 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-serving-cert\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:48:07.985386 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.985380 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-config\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:48:07.985386 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.985389 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-oauth-serving-cert\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:48:07.985386 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.985397 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk96v\" (UniqueName: \"kubernetes.io/projected/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-kube-api-access-qk96v\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:48:07.985636 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.985407 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-console-oauth-config\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:48:07.985636 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:07.985416 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/719cb6c1-7370-4e96-bd1a-6701eec6bb6a-service-ca\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:48:08.151240 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:08.151208 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59fb8745d5-wb5dk"] Jul 14 19:48:08.158325 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:08.158300 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59fb8745d5-wb5dk"] Jul 14 19:48:08.360636 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:08.360604 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="719cb6c1-7370-4e96-bd1a-6701eec6bb6a" path="/var/lib/kubelet/pods/719cb6c1-7370-4e96-bd1a-6701eec6bb6a/volumes" Jul 14 19:48:29.904647 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:29.904582 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5888c58fc-qzxlt" podUID="d7742d52-e657-447d-a209-61d317713990" containerName="console" containerID="cri-o://00a917726ebf827ec90b8ad05213c608f0258f0e1a0592ba89d2befee794f848" gracePeriod=15 Jul 14 19:48:30.140876 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.140853 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:48:30.259237 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.259155 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7742d52-e657-447d-a209-61d317713990-console-serving-cert\") pod \"d7742d52-e657-447d-a209-61d317713990\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " Jul 14 19:48:30.259237 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.259201 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7742d52-e657-447d-a209-61d317713990-console-oauth-config\") pod \"d7742d52-e657-447d-a209-61d317713990\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " Jul 14 19:48:30.259237 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.259221 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-oauth-serving-cert\") pod \"d7742d52-e657-447d-a209-61d317713990\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " Jul 14 19:48:30.259466 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.259240 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-console-config\") pod \"d7742d52-e657-447d-a209-61d317713990\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " Jul 14 19:48:30.259466 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.259452 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6bdp\" (UniqueName: \"kubernetes.io/projected/d7742d52-e657-447d-a209-61d317713990-kube-api-access-v6bdp\") pod \"d7742d52-e657-447d-a209-61d317713990\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " Jul 14 19:48:30.259543 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.259515 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-service-ca\") pod \"d7742d52-e657-447d-a209-61d317713990\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " Jul 14 19:48:30.259580 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.259570 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-trusted-ca-bundle\") pod \"d7742d52-e657-447d-a209-61d317713990\" (UID: \"d7742d52-e657-447d-a209-61d317713990\") " Jul 14 19:48:30.259653 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.259631 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-console-config" (OuterVolumeSpecName: "console-config") pod "d7742d52-e657-447d-a209-61d317713990" (UID: "d7742d52-e657-447d-a209-61d317713990"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:48:30.259653 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.259639 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d7742d52-e657-447d-a209-61d317713990" (UID: "d7742d52-e657-447d-a209-61d317713990"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:48:30.259862 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.259814 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-oauth-serving-cert\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:48:30.259862 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.259838 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-console-config\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:48:30.260034 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.259957 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-service-ca" (OuterVolumeSpecName: "service-ca") pod "d7742d52-e657-447d-a209-61d317713990" (UID: "d7742d52-e657-447d-a209-61d317713990"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:48:30.260128 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.260039 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d7742d52-e657-447d-a209-61d317713990" (UID: "d7742d52-e657-447d-a209-61d317713990"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:48:30.261520 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.261495 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7742d52-e657-447d-a209-61d317713990-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d7742d52-e657-447d-a209-61d317713990" (UID: "d7742d52-e657-447d-a209-61d317713990"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:48:30.261520 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.261505 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7742d52-e657-447d-a209-61d317713990-kube-api-access-v6bdp" (OuterVolumeSpecName: "kube-api-access-v6bdp") pod "d7742d52-e657-447d-a209-61d317713990" (UID: "d7742d52-e657-447d-a209-61d317713990"). InnerVolumeSpecName "kube-api-access-v6bdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:48:30.261654 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.261523 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7742d52-e657-447d-a209-61d317713990-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d7742d52-e657-447d-a209-61d317713990" (UID: "d7742d52-e657-447d-a209-61d317713990"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:48:30.360849 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.360819 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-service-ca\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:48:30.360849 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.360845 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7742d52-e657-447d-a209-61d317713990-trusted-ca-bundle\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:48:30.360849 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.360855 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7742d52-e657-447d-a209-61d317713990-console-serving-cert\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:48:30.361089 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.360863 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7742d52-e657-447d-a209-61d317713990-console-oauth-config\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:48:30.361089 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.360884 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6bdp\" (UniqueName: \"kubernetes.io/projected/d7742d52-e657-447d-a209-61d317713990-kube-api-access-v6bdp\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:48:30.919956 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.919919 2557 generic.go:356] "Generic (PLEG): container finished" podID="d7742d52-e657-447d-a209-61d317713990" containerID="00a917726ebf827ec90b8ad05213c608f0258f0e1a0592ba89d2befee794f848" exitCode=0 Jul 14 19:48:30.920431 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.919973 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5888c58fc-qzxlt" Jul 14 19:48:30.920431 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.919986 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5888c58fc-qzxlt" event={"ID":"d7742d52-e657-447d-a209-61d317713990","Type":"ContainerDied","Data":"00a917726ebf827ec90b8ad05213c608f0258f0e1a0592ba89d2befee794f848"} Jul 14 19:48:30.920431 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.920038 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5888c58fc-qzxlt" event={"ID":"d7742d52-e657-447d-a209-61d317713990","Type":"ContainerDied","Data":"293e34edb2656db2c170993928968f7bbcc03e86910dbb66d10d8675e1673a6a"} Jul 14 19:48:30.920431 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.920091 2557 scope.go:117] "RemoveContainer" containerID="00a917726ebf827ec90b8ad05213c608f0258f0e1a0592ba89d2befee794f848" Jul 14 19:48:30.928074 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.928040 2557 scope.go:117] "RemoveContainer" containerID="00a917726ebf827ec90b8ad05213c608f0258f0e1a0592ba89d2befee794f848" Jul 14 19:48:30.928325 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:48:30.928305 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00a917726ebf827ec90b8ad05213c608f0258f0e1a0592ba89d2befee794f848\": container with ID starting with 00a917726ebf827ec90b8ad05213c608f0258f0e1a0592ba89d2befee794f848 not found: ID does not exist" containerID="00a917726ebf827ec90b8ad05213c608f0258f0e1a0592ba89d2befee794f848" Jul 14 19:48:30.928373 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.928332 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00a917726ebf827ec90b8ad05213c608f0258f0e1a0592ba89d2befee794f848"} err="failed to get container status \"00a917726ebf827ec90b8ad05213c608f0258f0e1a0592ba89d2befee794f848\": rpc error: code = NotFound desc = could not find container \"00a917726ebf827ec90b8ad05213c608f0258f0e1a0592ba89d2befee794f848\": container with ID starting with 00a917726ebf827ec90b8ad05213c608f0258f0e1a0592ba89d2befee794f848 not found: ID does not exist" Jul 14 19:48:30.938892 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.938866 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5888c58fc-qzxlt"] Jul 14 19:48:30.946472 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:30.946449 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5888c58fc-qzxlt"] Jul 14 19:48:32.359560 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:32.359530 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7742d52-e657-447d-a209-61d317713990" path="/var/lib/kubelet/pods/d7742d52-e657-447d-a209-61d317713990/volumes" Jul 14 19:48:49.363140 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:49.363104 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:48:49.383043 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:49.383018 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:48:50.004349 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:48:50.004324 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 19:49:10.797388 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:10.797352 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6894598bd5-p94qv"] Jul 14 19:49:35.817519 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:35.817474 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6894598bd5-p94qv" podUID="8e649768-f5f6-4226-93ef-4123627698a1" containerName="console" containerID="cri-o://515cdc2c5bebecb480848974ee79d5b9bd352c27bc347e471adf7b7d07cc7a32" gracePeriod=15 Jul 14 19:49:36.059796 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.059771 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:49:36.159991 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.159959 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-oauth-serving-cert\") pod \"8e649768-f5f6-4226-93ef-4123627698a1\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " Jul 14 19:49:36.160207 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.160001 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-console-config\") pod \"8e649768-f5f6-4226-93ef-4123627698a1\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " Jul 14 19:49:36.160207 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.160022 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e649768-f5f6-4226-93ef-4123627698a1-console-serving-cert\") pod \"8e649768-f5f6-4226-93ef-4123627698a1\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " Jul 14 19:49:36.160207 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.160158 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlr65\" (UniqueName: \"kubernetes.io/projected/8e649768-f5f6-4226-93ef-4123627698a1-kube-api-access-jlr65\") pod \"8e649768-f5f6-4226-93ef-4123627698a1\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " Jul 14 19:49:36.160432 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.160212 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-service-ca\") pod \"8e649768-f5f6-4226-93ef-4123627698a1\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " Jul 14 19:49:36.160432 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.160260 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-trusted-ca-bundle\") pod \"8e649768-f5f6-4226-93ef-4123627698a1\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " Jul 14 19:49:36.160432 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.160300 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e649768-f5f6-4226-93ef-4123627698a1-console-oauth-config\") pod \"8e649768-f5f6-4226-93ef-4123627698a1\" (UID: \"8e649768-f5f6-4226-93ef-4123627698a1\") " Jul 14 19:49:36.160583 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.160442 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-console-config" (OuterVolumeSpecName: "console-config") pod "8e649768-f5f6-4226-93ef-4123627698a1" (UID: "8e649768-f5f6-4226-93ef-4123627698a1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:49:36.160583 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.160439 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8e649768-f5f6-4226-93ef-4123627698a1" (UID: "8e649768-f5f6-4226-93ef-4123627698a1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:49:36.160710 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.160679 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-service-ca" (OuterVolumeSpecName: "service-ca") pod "8e649768-f5f6-4226-93ef-4123627698a1" (UID: "8e649768-f5f6-4226-93ef-4123627698a1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:49:36.160827 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.160767 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8e649768-f5f6-4226-93ef-4123627698a1" (UID: "8e649768-f5f6-4226-93ef-4123627698a1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:49:36.161134 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.161109 2557 generic.go:356] "Generic (PLEG): container finished" podID="8e649768-f5f6-4226-93ef-4123627698a1" containerID="515cdc2c5bebecb480848974ee79d5b9bd352c27bc347e471adf7b7d07cc7a32" exitCode=0 Jul 14 19:49:36.161220 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.161177 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6894598bd5-p94qv" Jul 14 19:49:36.161271 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.161177 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6894598bd5-p94qv" event={"ID":"8e649768-f5f6-4226-93ef-4123627698a1","Type":"ContainerDied","Data":"515cdc2c5bebecb480848974ee79d5b9bd352c27bc347e471adf7b7d07cc7a32"} Jul 14 19:49:36.161324 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.161303 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6894598bd5-p94qv" event={"ID":"8e649768-f5f6-4226-93ef-4123627698a1","Type":"ContainerDied","Data":"fcd396725ca7ac1818d6510e0cbabf3dffda28abc893faa97ed11041155aa8b9"} Jul 14 19:49:36.161374 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.161334 2557 scope.go:117] "RemoveContainer" containerID="515cdc2c5bebecb480848974ee79d5b9bd352c27bc347e471adf7b7d07cc7a32" Jul 14 19:49:36.162493 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.162465 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e649768-f5f6-4226-93ef-4123627698a1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8e649768-f5f6-4226-93ef-4123627698a1" (UID: "8e649768-f5f6-4226-93ef-4123627698a1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:49:36.162670 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.162638 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e649768-f5f6-4226-93ef-4123627698a1-kube-api-access-jlr65" (OuterVolumeSpecName: "kube-api-access-jlr65") pod "8e649768-f5f6-4226-93ef-4123627698a1" (UID: "8e649768-f5f6-4226-93ef-4123627698a1"). InnerVolumeSpecName "kube-api-access-jlr65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:49:36.162725 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.162699 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e649768-f5f6-4226-93ef-4123627698a1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8e649768-f5f6-4226-93ef-4123627698a1" (UID: "8e649768-f5f6-4226-93ef-4123627698a1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:49:36.175685 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.175664 2557 scope.go:117] "RemoveContainer" containerID="515cdc2c5bebecb480848974ee79d5b9bd352c27bc347e471adf7b7d07cc7a32" Jul 14 19:49:36.175970 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:49:36.175951 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515cdc2c5bebecb480848974ee79d5b9bd352c27bc347e471adf7b7d07cc7a32\": container with ID starting with 515cdc2c5bebecb480848974ee79d5b9bd352c27bc347e471adf7b7d07cc7a32 not found: ID does not exist" containerID="515cdc2c5bebecb480848974ee79d5b9bd352c27bc347e471adf7b7d07cc7a32" Jul 14 19:49:36.176037 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.175983 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515cdc2c5bebecb480848974ee79d5b9bd352c27bc347e471adf7b7d07cc7a32"} err="failed to get container status \"515cdc2c5bebecb480848974ee79d5b9bd352c27bc347e471adf7b7d07cc7a32\": rpc error: code = NotFound desc = could not find container \"515cdc2c5bebecb480848974ee79d5b9bd352c27bc347e471adf7b7d07cc7a32\": container with ID starting with 515cdc2c5bebecb480848974ee79d5b9bd352c27bc347e471adf7b7d07cc7a32 not found: ID does not exist" Jul 14 19:49:36.260908 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.260877 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-oauth-serving-cert\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:49:36.260908 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.260904 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-console-config\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:49:36.260908 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.260913 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e649768-f5f6-4226-93ef-4123627698a1-console-serving-cert\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:49:36.261177 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.260922 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlr65\" (UniqueName: \"kubernetes.io/projected/8e649768-f5f6-4226-93ef-4123627698a1-kube-api-access-jlr65\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:49:36.261177 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.260930 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-service-ca\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:49:36.261177 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.260938 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e649768-f5f6-4226-93ef-4123627698a1-trusted-ca-bundle\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:49:36.261177 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.260946 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e649768-f5f6-4226-93ef-4123627698a1-console-oauth-config\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:49:36.479810 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.479727 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6894598bd5-p94qv"] Jul 14 19:49:36.483398 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:36.483364 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6894598bd5-p94qv"] Jul 14 19:49:38.359289 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:49:38.359255 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e649768-f5f6-4226-93ef-4123627698a1" path="/var/lib/kubelet/pods/8e649768-f5f6-4226-93ef-4123627698a1/volumes" Jul 14 19:50:51.491846 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.491772 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-q9r77"] Jul 14 19:50:51.492302 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.492111 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7742d52-e657-447d-a209-61d317713990" containerName="console" Jul 14 19:50:51.492302 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.492127 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7742d52-e657-447d-a209-61d317713990" containerName="console" Jul 14 19:50:51.492302 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.492140 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e649768-f5f6-4226-93ef-4123627698a1" containerName="console" Jul 14 19:50:51.492302 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.492146 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e649768-f5f6-4226-93ef-4123627698a1" containerName="console" Jul 14 19:50:51.492302 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.492159 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="719cb6c1-7370-4e96-bd1a-6701eec6bb6a" containerName="console" Jul 14 19:50:51.492302 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.492167 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="719cb6c1-7370-4e96-bd1a-6701eec6bb6a" containerName="console" Jul 14 19:50:51.492302 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.492227 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e649768-f5f6-4226-93ef-4123627698a1" containerName="console" Jul 14 19:50:51.492302 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.492234 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7742d52-e657-447d-a209-61d317713990" containerName="console" Jul 14 19:50:51.492302 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.492244 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="719cb6c1-7370-4e96-bd1a-6701eec6bb6a" containerName="console" Jul 14 19:50:51.494125 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.494110 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q9r77" Jul 14 19:50:51.496663 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.496638 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-wx9kz\"" Jul 14 19:50:51.496787 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.496665 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 14 19:50:51.497207 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.497189 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 14 19:50:51.505764 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.505740 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-q9r77"] Jul 14 19:50:51.536649 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.536614 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj522\" (UniqueName: \"kubernetes.io/projected/b72e2c48-b796-488e-b4c8-1dea282aee0f-kube-api-access-vj522\") pod \"cert-manager-cainjector-559cdd8c87-q9r77\" (UID: \"b72e2c48-b796-488e-b4c8-1dea282aee0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q9r77" Jul 14 19:50:51.536649 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.536654 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b72e2c48-b796-488e-b4c8-1dea282aee0f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-q9r77\" (UID: \"b72e2c48-b796-488e-b4c8-1dea282aee0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q9r77" Jul 14 19:50:51.637969 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.637929 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj522\" (UniqueName: \"kubernetes.io/projected/b72e2c48-b796-488e-b4c8-1dea282aee0f-kube-api-access-vj522\") pod \"cert-manager-cainjector-559cdd8c87-q9r77\" (UID: \"b72e2c48-b796-488e-b4c8-1dea282aee0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q9r77" Jul 14 19:50:51.637969 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.637976 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b72e2c48-b796-488e-b4c8-1dea282aee0f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-q9r77\" (UID: \"b72e2c48-b796-488e-b4c8-1dea282aee0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q9r77" Jul 14 19:50:51.650369 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.650339 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b72e2c48-b796-488e-b4c8-1dea282aee0f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-q9r77\" (UID: \"b72e2c48-b796-488e-b4c8-1dea282aee0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q9r77" Jul 14 19:50:51.650494 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.650474 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj522\" (UniqueName: \"kubernetes.io/projected/b72e2c48-b796-488e-b4c8-1dea282aee0f-kube-api-access-vj522\") pod \"cert-manager-cainjector-559cdd8c87-q9r77\" (UID: \"b72e2c48-b796-488e-b4c8-1dea282aee0f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-q9r77" Jul 14 19:50:51.811964 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.811864 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q9r77" Jul 14 19:50:51.937362 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:51.937338 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-q9r77"] Jul 14 19:50:51.939029 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:50:51.938997 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb72e2c48_b796_488e_b4c8_1dea282aee0f.slice/crio-88234b8f239ed4941be57e74fb0d1c949d1798e586224508f64d142c333c13b9 WatchSource:0}: Error finding container 88234b8f239ed4941be57e74fb0d1c949d1798e586224508f64d142c333c13b9: Status 404 returned error can't find the container with id 88234b8f239ed4941be57e74fb0d1c949d1798e586224508f64d142c333c13b9 Jul 14 19:50:52.420039 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:52.420004 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q9r77" event={"ID":"b72e2c48-b796-488e-b4c8-1dea282aee0f","Type":"ContainerStarted","Data":"88234b8f239ed4941be57e74fb0d1c949d1798e586224508f64d142c333c13b9"} Jul 14 19:50:55.433476 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:55.433437 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q9r77" event={"ID":"b72e2c48-b796-488e-b4c8-1dea282aee0f","Type":"ContainerStarted","Data":"765a926ef0b2750d2cb8e2c7a64a69cfcc87357d5bee115205fff202ab88bc5e"} Jul 14 19:50:55.458615 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:50:55.458562 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-q9r77" podStartSLOduration=1.3529411740000001 podStartE2EDuration="4.458544348s" totalImagesPullingTime="3.105603174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:50:55.458117701 +0000 UTC m=+273.537863517" watchObservedRunningTime="2026-07-14 19:50:55.458544348 +0000 UTC m=+273.538290142" Jul 14 19:51:09.168558 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:09.168520 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-59zbk"] Jul 14 19:51:09.170526 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:09.170510 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-59zbk" Jul 14 19:51:09.173221 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:09.173199 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-p9ppm\"" Jul 14 19:51:09.181021 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:09.180997 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-59zbk"] Jul 14 19:51:09.277882 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:09.277843 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhlcj\" (UniqueName: \"kubernetes.io/projected/9f90702f-5498-4478-9d73-77f4a2d8d555-kube-api-access-dhlcj\") pod \"cert-manager-55b4d954d5-59zbk\" (UID: \"9f90702f-5498-4478-9d73-77f4a2d8d555\") " pod="cert-manager/cert-manager-55b4d954d5-59zbk" Jul 14 19:51:09.278049 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:09.277899 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f90702f-5498-4478-9d73-77f4a2d8d555-bound-sa-token\") pod \"cert-manager-55b4d954d5-59zbk\" (UID: \"9f90702f-5498-4478-9d73-77f4a2d8d555\") " pod="cert-manager/cert-manager-55b4d954d5-59zbk" Jul 14 19:51:09.379035 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:09.379003 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhlcj\" (UniqueName: \"kubernetes.io/projected/9f90702f-5498-4478-9d73-77f4a2d8d555-kube-api-access-dhlcj\") pod \"cert-manager-55b4d954d5-59zbk\" (UID: \"9f90702f-5498-4478-9d73-77f4a2d8d555\") " pod="cert-manager/cert-manager-55b4d954d5-59zbk" Jul 14 19:51:09.379203 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:09.379070 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f90702f-5498-4478-9d73-77f4a2d8d555-bound-sa-token\") pod \"cert-manager-55b4d954d5-59zbk\" (UID: \"9f90702f-5498-4478-9d73-77f4a2d8d555\") " pod="cert-manager/cert-manager-55b4d954d5-59zbk" Jul 14 19:51:09.391289 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:09.391263 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f90702f-5498-4478-9d73-77f4a2d8d555-bound-sa-token\") pod \"cert-manager-55b4d954d5-59zbk\" (UID: \"9f90702f-5498-4478-9d73-77f4a2d8d555\") " pod="cert-manager/cert-manager-55b4d954d5-59zbk" Jul 14 19:51:09.391289 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:09.391283 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhlcj\" (UniqueName: \"kubernetes.io/projected/9f90702f-5498-4478-9d73-77f4a2d8d555-kube-api-access-dhlcj\") pod \"cert-manager-55b4d954d5-59zbk\" (UID: \"9f90702f-5498-4478-9d73-77f4a2d8d555\") " pod="cert-manager/cert-manager-55b4d954d5-59zbk" Jul 14 19:51:09.478836 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:09.478750 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-59zbk" Jul 14 19:51:09.602933 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:09.602863 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-59zbk"] Jul 14 19:51:09.605716 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:51:09.605686 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f90702f_5498_4478_9d73_77f4a2d8d555.slice/crio-a72bf62e099e6374fa13f2263a6bb33626186c60e3896b8d7250298eeef15a99 WatchSource:0}: Error finding container a72bf62e099e6374fa13f2263a6bb33626186c60e3896b8d7250298eeef15a99: Status 404 returned error can't find the container with id a72bf62e099e6374fa13f2263a6bb33626186c60e3896b8d7250298eeef15a99 Jul 14 19:51:10.485918 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:10.485886 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-59zbk" event={"ID":"9f90702f-5498-4478-9d73-77f4a2d8d555","Type":"ContainerStarted","Data":"7ce4937a5f7024334bf43dc940b778aaf33d567b3e255c7b18bdad4e8f54df8d"} Jul 14 19:51:10.486325 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:10.485927 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-59zbk" event={"ID":"9f90702f-5498-4478-9d73-77f4a2d8d555","Type":"ContainerStarted","Data":"a72bf62e099e6374fa13f2263a6bb33626186c60e3896b8d7250298eeef15a99"} Jul 14 19:51:10.503545 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:10.503501 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-59zbk" podStartSLOduration=1.503486224 podStartE2EDuration="1.503486224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:51:10.502592976 +0000 UTC m=+288.582338789" watchObservedRunningTime="2026-07-14 19:51:10.503486224 +0000 UTC m=+288.583232016" Jul 14 19:51:22.262185 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:22.262153 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 19:51:22.266557 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:22.266533 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 19:51:22.267248 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:22.267230 2557 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 19:51:26.403596 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.403565 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp"] Jul 14 19:51:26.411067 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.411042 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.413751 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.413718 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 19:51:26.413875 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.413842 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 19:51:26.413946 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.413896 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5l9cj\"" Jul 14 19:51:26.414228 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.414207 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 19:51:26.414396 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.414374 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 19:51:26.425340 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.425316 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp"] Jul 14 19:51:26.516366 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.516330 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cccba306-e43d-4a4b-be0d-9b4a462f463e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-652mp\" (UID: \"cccba306-e43d-4a4b-be0d-9b4a462f463e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.516366 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.516369 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cccba306-e43d-4a4b-be0d-9b4a462f463e-manifests\") pod \"opendatahub-operator-controller-manager-7f59db9768-652mp\" (UID: \"cccba306-e43d-4a4b-be0d-9b4a462f463e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.516578 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.516389 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7bx8\" (UniqueName: \"kubernetes.io/projected/cccba306-e43d-4a4b-be0d-9b4a462f463e-kube-api-access-m7bx8\") pod \"opendatahub-operator-controller-manager-7f59db9768-652mp\" (UID: \"cccba306-e43d-4a4b-be0d-9b4a462f463e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.516578 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.516463 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cccba306-e43d-4a4b-be0d-9b4a462f463e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-652mp\" (UID: \"cccba306-e43d-4a4b-be0d-9b4a462f463e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.516578 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.516529 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cccba306-e43d-4a4b-be0d-9b4a462f463e-tmp\") pod \"opendatahub-operator-controller-manager-7f59db9768-652mp\" (UID: \"cccba306-e43d-4a4b-be0d-9b4a462f463e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.617911 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.617876 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cccba306-e43d-4a4b-be0d-9b4a462f463e-manifests\") pod \"opendatahub-operator-controller-manager-7f59db9768-652mp\" (UID: \"cccba306-e43d-4a4b-be0d-9b4a462f463e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.617911 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.617911 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7bx8\" (UniqueName: \"kubernetes.io/projected/cccba306-e43d-4a4b-be0d-9b4a462f463e-kube-api-access-m7bx8\") pod \"opendatahub-operator-controller-manager-7f59db9768-652mp\" (UID: \"cccba306-e43d-4a4b-be0d-9b4a462f463e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.618153 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.617940 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cccba306-e43d-4a4b-be0d-9b4a462f463e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-652mp\" (UID: \"cccba306-e43d-4a4b-be0d-9b4a462f463e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.618153 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.618000 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cccba306-e43d-4a4b-be0d-9b4a462f463e-tmp\") pod \"opendatahub-operator-controller-manager-7f59db9768-652mp\" (UID: \"cccba306-e43d-4a4b-be0d-9b4a462f463e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.618153 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.618052 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cccba306-e43d-4a4b-be0d-9b4a462f463e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-652mp\" (UID: \"cccba306-e43d-4a4b-be0d-9b4a462f463e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.618362 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.618339 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cccba306-e43d-4a4b-be0d-9b4a462f463e-manifests\") pod \"opendatahub-operator-controller-manager-7f59db9768-652mp\" (UID: \"cccba306-e43d-4a4b-be0d-9b4a462f463e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.620422 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.620389 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cccba306-e43d-4a4b-be0d-9b4a462f463e-tmp\") pod \"opendatahub-operator-controller-manager-7f59db9768-652mp\" (UID: \"cccba306-e43d-4a4b-be0d-9b4a462f463e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.620532 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.620428 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cccba306-e43d-4a4b-be0d-9b4a462f463e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-652mp\" (UID: \"cccba306-e43d-4a4b-be0d-9b4a462f463e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.620532 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.620467 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cccba306-e43d-4a4b-be0d-9b4a462f463e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-652mp\" (UID: \"cccba306-e43d-4a4b-be0d-9b4a462f463e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.628435 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.628407 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7bx8\" (UniqueName: \"kubernetes.io/projected/cccba306-e43d-4a4b-be0d-9b4a462f463e-kube-api-access-m7bx8\") pod \"opendatahub-operator-controller-manager-7f59db9768-652mp\" (UID: \"cccba306-e43d-4a4b-be0d-9b4a462f463e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.723255 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.723169 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:26.858102 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.858075 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp"] Jul 14 19:51:26.860587 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:51:26.860564 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccba306_e43d_4a4b_be0d_9b4a462f463e.slice/crio-18b86980900dbbb7eb1a080c87367f59db2c751f935674d6f51720535ae25bb4 WatchSource:0}: Error finding container 18b86980900dbbb7eb1a080c87367f59db2c751f935674d6f51720535ae25bb4: Status 404 returned error can't find the container with id 18b86980900dbbb7eb1a080c87367f59db2c751f935674d6f51720535ae25bb4 Jul 14 19:51:26.862439 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:26.862423 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:51:27.546136 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:27.546102 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" event={"ID":"cccba306-e43d-4a4b-be0d-9b4a462f463e","Type":"ContainerStarted","Data":"18b86980900dbbb7eb1a080c87367f59db2c751f935674d6f51720535ae25bb4"} Jul 14 19:51:30.561780 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:30.561748 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" event={"ID":"cccba306-e43d-4a4b-be0d-9b4a462f463e","Type":"ContainerStarted","Data":"89c95ffba36a482552c28fbfb8b00c7c51846de9846c74c434a77f7d3aff18a3"} Jul 14 19:51:31.566967 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:31.566870 2557 generic.go:356] "Generic (PLEG): container finished" podID="cccba306-e43d-4a4b-be0d-9b4a462f463e" containerID="89c95ffba36a482552c28fbfb8b00c7c51846de9846c74c434a77f7d3aff18a3" exitCode=0 Jul 14 19:51:31.566967 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:31.566922 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" event={"ID":"cccba306-e43d-4a4b-be0d-9b4a462f463e","Type":"ContainerDied","Data":"89c95ffba36a482552c28fbfb8b00c7c51846de9846c74c434a77f7d3aff18a3"} Jul 14 19:51:32.574514 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:32.574481 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" event={"ID":"cccba306-e43d-4a4b-be0d-9b4a462f463e","Type":"ContainerStarted","Data":"315db3317b6c0e9c0a40db4b3bdb53db784386b4ca9f5fa84d1e6b4035a71d59"} Jul 14 19:51:32.574907 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:32.574635 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:32.596001 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:32.595950 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" podStartSLOduration=1.963071068 podStartE2EDuration="6.595933471s" totalImagesPullingTime="3.608501134s" totalInitContainerRuntime="1.024361269s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:51:32.594926 +0000 UTC m=+310.674671809" watchObservedRunningTime="2026-07-14 19:51:32.595933471 +0000 UTC m=+310.675679264" Jul 14 19:51:43.580129 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:43.580097 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-652mp" Jul 14 19:51:44.492344 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.492305 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x"] Jul 14 19:51:44.495409 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.495385 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:44.498311 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.498285 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 14 19:51:44.498430 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.498393 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 14 19:51:44.498430 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.498392 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fh6nk\"" Jul 14 19:51:44.498541 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.498495 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:51:44.498630 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.498611 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 19:51:44.498683 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.498613 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 14 19:51:44.506549 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.506528 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x"] Jul 14 19:51:44.565932 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.565904 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9f6db25a-56f4-4636-b6e4-56c351a834c9-manager-config\") pod \"lws-controller-manager-5dc78d48f4-2ft4x\" (UID: \"9f6db25a-56f4-4636-b6e4-56c351a834c9\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:44.566132 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.565940 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f6db25a-56f4-4636-b6e4-56c351a834c9-metrics-cert\") pod \"lws-controller-manager-5dc78d48f4-2ft4x\" (UID: \"9f6db25a-56f4-4636-b6e4-56c351a834c9\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:44.566132 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.566041 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f6db25a-56f4-4636-b6e4-56c351a834c9-cert\") pod \"lws-controller-manager-5dc78d48f4-2ft4x\" (UID: \"9f6db25a-56f4-4636-b6e4-56c351a834c9\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:44.566132 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.566102 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvz6g\" (UniqueName: \"kubernetes.io/projected/9f6db25a-56f4-4636-b6e4-56c351a834c9-kube-api-access-rvz6g\") pod \"lws-controller-manager-5dc78d48f4-2ft4x\" (UID: \"9f6db25a-56f4-4636-b6e4-56c351a834c9\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:44.667076 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.667018 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f6db25a-56f4-4636-b6e4-56c351a834c9-cert\") pod \"lws-controller-manager-5dc78d48f4-2ft4x\" (UID: \"9f6db25a-56f4-4636-b6e4-56c351a834c9\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:44.667523 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.667092 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvz6g\" (UniqueName: \"kubernetes.io/projected/9f6db25a-56f4-4636-b6e4-56c351a834c9-kube-api-access-rvz6g\") pod \"lws-controller-manager-5dc78d48f4-2ft4x\" (UID: \"9f6db25a-56f4-4636-b6e4-56c351a834c9\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:44.667523 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.667162 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9f6db25a-56f4-4636-b6e4-56c351a834c9-manager-config\") pod \"lws-controller-manager-5dc78d48f4-2ft4x\" (UID: \"9f6db25a-56f4-4636-b6e4-56c351a834c9\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:44.667523 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.667185 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f6db25a-56f4-4636-b6e4-56c351a834c9-metrics-cert\") pod \"lws-controller-manager-5dc78d48f4-2ft4x\" (UID: \"9f6db25a-56f4-4636-b6e4-56c351a834c9\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:44.667922 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.667899 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9f6db25a-56f4-4636-b6e4-56c351a834c9-manager-config\") pod \"lws-controller-manager-5dc78d48f4-2ft4x\" (UID: \"9f6db25a-56f4-4636-b6e4-56c351a834c9\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:44.669656 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.669637 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f6db25a-56f4-4636-b6e4-56c351a834c9-cert\") pod \"lws-controller-manager-5dc78d48f4-2ft4x\" (UID: \"9f6db25a-56f4-4636-b6e4-56c351a834c9\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:44.669713 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.669663 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f6db25a-56f4-4636-b6e4-56c351a834c9-metrics-cert\") pod \"lws-controller-manager-5dc78d48f4-2ft4x\" (UID: \"9f6db25a-56f4-4636-b6e4-56c351a834c9\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:44.676105 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.676081 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvz6g\" (UniqueName: \"kubernetes.io/projected/9f6db25a-56f4-4636-b6e4-56c351a834c9-kube-api-access-rvz6g\") pod \"lws-controller-manager-5dc78d48f4-2ft4x\" (UID: \"9f6db25a-56f4-4636-b6e4-56c351a834c9\") " pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:44.804715 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.804628 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:44.933890 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:44.933861 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x"] Jul 14 19:51:44.935819 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:51:44.935790 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6db25a_56f4_4636_b6e4_56c351a834c9.slice/crio-c3ebe934efb2999dec72fe503831ab8d245e74e9292896e5b5cc214f128983fe WatchSource:0}: Error finding container c3ebe934efb2999dec72fe503831ab8d245e74e9292896e5b5cc214f128983fe: Status 404 returned error can't find the container with id c3ebe934efb2999dec72fe503831ab8d245e74e9292896e5b5cc214f128983fe Jul 14 19:51:45.623387 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:45.623350 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" event={"ID":"9f6db25a-56f4-4636-b6e4-56c351a834c9","Type":"ContainerStarted","Data":"c3ebe934efb2999dec72fe503831ab8d245e74e9292896e5b5cc214f128983fe"} Jul 14 19:51:48.642386 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:48.642350 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" event={"ID":"9f6db25a-56f4-4636-b6e4-56c351a834c9","Type":"ContainerStarted","Data":"3ff604ed20b2e1788950ec7248d5042f31a158306bd61db59d7d90e78f51ad79"} Jul 14 19:51:48.642872 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:48.642430 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:48.664937 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:48.664878 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" podStartSLOduration=1.609732631 podStartE2EDuration="4.664862077s" totalImagesPullingTime="3.055129446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:51:48.663659517 +0000 UTC m=+326.743405333" watchObservedRunningTime="2026-07-14 19:51:48.664862077 +0000 UTC m=+326.744607868" Jul 14 19:51:58.235145 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.235111 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf"] Jul 14 19:51:58.239715 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.239694 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf" Jul 14 19:51:58.243951 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.243925 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 19:51:58.244086 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.243974 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hl25t\"" Jul 14 19:51:58.244588 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.244575 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 14 19:51:58.244684 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.244669 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 14 19:51:58.246113 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.246099 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 19:51:58.259957 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.259922 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf"] Jul 14 19:51:58.280973 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.280936 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eef7830-07e1-4a2c-9734-fb9d39eb4c13-tls-certs\") pod \"kube-auth-proxy-5df66fb8c-s42jf\" (UID: \"5eef7830-07e1-4a2c-9734-fb9d39eb4c13\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf" Jul 14 19:51:58.281148 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.281009 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8x9q\" (UniqueName: \"kubernetes.io/projected/5eef7830-07e1-4a2c-9734-fb9d39eb4c13-kube-api-access-h8x9q\") pod \"kube-auth-proxy-5df66fb8c-s42jf\" (UID: \"5eef7830-07e1-4a2c-9734-fb9d39eb4c13\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf" Jul 14 19:51:58.281148 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.281095 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eef7830-07e1-4a2c-9734-fb9d39eb4c13-tmp\") pod \"kube-auth-proxy-5df66fb8c-s42jf\" (UID: \"5eef7830-07e1-4a2c-9734-fb9d39eb4c13\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf" Jul 14 19:51:58.381608 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.381565 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eef7830-07e1-4a2c-9734-fb9d39eb4c13-tls-certs\") pod \"kube-auth-proxy-5df66fb8c-s42jf\" (UID: \"5eef7830-07e1-4a2c-9734-fb9d39eb4c13\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf" Jul 14 19:51:58.381775 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.381624 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8x9q\" (UniqueName: \"kubernetes.io/projected/5eef7830-07e1-4a2c-9734-fb9d39eb4c13-kube-api-access-h8x9q\") pod \"kube-auth-proxy-5df66fb8c-s42jf\" (UID: \"5eef7830-07e1-4a2c-9734-fb9d39eb4c13\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf" Jul 14 19:51:58.381775 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.381656 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eef7830-07e1-4a2c-9734-fb9d39eb4c13-tmp\") pod \"kube-auth-proxy-5df66fb8c-s42jf\" (UID: \"5eef7830-07e1-4a2c-9734-fb9d39eb4c13\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf" Jul 14 19:51:58.381775 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:51:58.381708 2557 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 14 19:51:58.381894 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:51:58.381777 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eef7830-07e1-4a2c-9734-fb9d39eb4c13-tls-certs podName:5eef7830-07e1-4a2c-9734-fb9d39eb4c13 nodeName:}" failed. No retries permitted until 2026-07-14 19:51:58.8817612 +0000 UTC m=+336.961506973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5eef7830-07e1-4a2c-9734-fb9d39eb4c13-tls-certs") pod "kube-auth-proxy-5df66fb8c-s42jf" (UID: "5eef7830-07e1-4a2c-9734-fb9d39eb4c13") : secret "kube-auth-proxy-tls" not found Jul 14 19:51:58.383834 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.383810 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5eef7830-07e1-4a2c-9734-fb9d39eb4c13-tmp\") pod \"kube-auth-proxy-5df66fb8c-s42jf\" (UID: \"5eef7830-07e1-4a2c-9734-fb9d39eb4c13\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf" Jul 14 19:51:58.391870 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.391848 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8x9q\" (UniqueName: \"kubernetes.io/projected/5eef7830-07e1-4a2c-9734-fb9d39eb4c13-kube-api-access-h8x9q\") pod \"kube-auth-proxy-5df66fb8c-s42jf\" (UID: \"5eef7830-07e1-4a2c-9734-fb9d39eb4c13\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf" Jul 14 19:51:58.885803 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.885759 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eef7830-07e1-4a2c-9734-fb9d39eb4c13-tls-certs\") pod \"kube-auth-proxy-5df66fb8c-s42jf\" (UID: \"5eef7830-07e1-4a2c-9734-fb9d39eb4c13\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf" Jul 14 19:51:58.888176 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:58.888143 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eef7830-07e1-4a2c-9734-fb9d39eb4c13-tls-certs\") pod \"kube-auth-proxy-5df66fb8c-s42jf\" (UID: \"5eef7830-07e1-4a2c-9734-fb9d39eb4c13\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf" Jul 14 19:51:59.149107 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:59.148996 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf" Jul 14 19:51:59.277672 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:59.277643 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf"] Jul 14 19:51:59.281145 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:51:59.281112 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eef7830_07e1_4a2c_9734_fb9d39eb4c13.slice/crio-fa9320b44ee8e8537d2ad9ed6ffe1887a1876a83070bd04b7b950e58f944f7f2 WatchSource:0}: Error finding container fa9320b44ee8e8537d2ad9ed6ffe1887a1876a83070bd04b7b950e58f944f7f2: Status 404 returned error can't find the container with id fa9320b44ee8e8537d2ad9ed6ffe1887a1876a83070bd04b7b950e58f944f7f2 Jul 14 19:51:59.649004 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:59.648972 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5dc78d48f4-2ft4x" Jul 14 19:51:59.686612 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:51:59.686569 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf" event={"ID":"5eef7830-07e1-4a2c-9734-fb9d39eb4c13","Type":"ContainerStarted","Data":"fa9320b44ee8e8537d2ad9ed6ffe1887a1876a83070bd04b7b950e58f944f7f2"} Jul 14 19:52:04.708422 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:04.708322 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf" event={"ID":"5eef7830-07e1-4a2c-9734-fb9d39eb4c13","Type":"ContainerStarted","Data":"e931de1bb35a671e587955d93e1fbaa22bf689ec66e7231e3f74e01138d49475"} Jul 14 19:52:04.737696 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:04.737627 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5df66fb8c-s42jf" podStartSLOduration=1.60064126 podStartE2EDuration="6.737605458s" totalImagesPullingTime="5.136964198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:04.736359905 +0000 UTC m=+342.816105735" watchObservedRunningTime="2026-07-14 19:52:04.737605458 +0000 UTC m=+342.817351255" Jul 14 19:52:59.105331 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.105300 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fbf959699-lg2wp"] Jul 14 19:52:59.108512 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.108496 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.110717 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.110698 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 19:52:59.110829 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.110809 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 19:52:59.111540 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.111510 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h5xm7\"" Jul 14 19:52:59.111655 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.111553 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 19:52:59.111655 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.111558 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 19:52:59.111655 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.111611 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 19:52:59.111798 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.111613 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 19:52:59.111798 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.111612 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 19:52:59.115870 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.115847 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 14 19:52:59.122155 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.122134 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbf959699-lg2wp"] Jul 14 19:52:59.189873 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.189840 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e70ccdd9-a7a2-441d-931f-38c86f0ad341-console-config\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.190050 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.189884 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e70ccdd9-a7a2-441d-931f-38c86f0ad341-trusted-ca-bundle\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.190050 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.189979 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e70ccdd9-a7a2-441d-931f-38c86f0ad341-oauth-serving-cert\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.190050 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.190017 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e70ccdd9-a7a2-441d-931f-38c86f0ad341-console-serving-cert\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.190247 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.190037 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e70ccdd9-a7a2-441d-931f-38c86f0ad341-service-ca\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.190592 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.190565 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lg2s\" (UniqueName: \"kubernetes.io/projected/e70ccdd9-a7a2-441d-931f-38c86f0ad341-kube-api-access-4lg2s\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.190685 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.190650 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e70ccdd9-a7a2-441d-931f-38c86f0ad341-console-oauth-config\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.291248 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.291192 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e70ccdd9-a7a2-441d-931f-38c86f0ad341-trusted-ca-bundle\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.291458 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.291280 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e70ccdd9-a7a2-441d-931f-38c86f0ad341-oauth-serving-cert\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.291458 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.291312 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e70ccdd9-a7a2-441d-931f-38c86f0ad341-console-serving-cert\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.291458 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.291329 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e70ccdd9-a7a2-441d-931f-38c86f0ad341-service-ca\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.291458 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.291346 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lg2s\" (UniqueName: \"kubernetes.io/projected/e70ccdd9-a7a2-441d-931f-38c86f0ad341-kube-api-access-4lg2s\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.291458 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.291383 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e70ccdd9-a7a2-441d-931f-38c86f0ad341-console-oauth-config\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.291458 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.291400 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e70ccdd9-a7a2-441d-931f-38c86f0ad341-console-config\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.292015 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.291986 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e70ccdd9-a7a2-441d-931f-38c86f0ad341-oauth-serving-cert\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.292160 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.292050 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e70ccdd9-a7a2-441d-931f-38c86f0ad341-console-config\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.292227 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.292186 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e70ccdd9-a7a2-441d-931f-38c86f0ad341-service-ca\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.292227 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.292192 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e70ccdd9-a7a2-441d-931f-38c86f0ad341-trusted-ca-bundle\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.293904 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.293884 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e70ccdd9-a7a2-441d-931f-38c86f0ad341-console-oauth-config\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.293990 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.293973 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e70ccdd9-a7a2-441d-931f-38c86f0ad341-console-serving-cert\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.300643 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.300620 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lg2s\" (UniqueName: \"kubernetes.io/projected/e70ccdd9-a7a2-441d-931f-38c86f0ad341-kube-api-access-4lg2s\") pod \"console-7fbf959699-lg2wp\" (UID: \"e70ccdd9-a7a2-441d-931f-38c86f0ad341\") " pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.417954 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.417873 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:52:59.547799 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.547776 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbf959699-lg2wp"] Jul 14 19:52:59.550015 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:52:59.549952 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70ccdd9_a7a2_441d_931f_38c86f0ad341.slice/crio-a97e90ae9cac761c3953515eae104820148375820697e7ed373dbf4b84cd2daf WatchSource:0}: Error finding container a97e90ae9cac761c3953515eae104820148375820697e7ed373dbf4b84cd2daf: Status 404 returned error can't find the container with id a97e90ae9cac761c3953515eae104820148375820697e7ed373dbf4b84cd2daf Jul 14 19:52:59.917137 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.917096 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbf959699-lg2wp" event={"ID":"e70ccdd9-a7a2-441d-931f-38c86f0ad341","Type":"ContainerStarted","Data":"ff56dfd5143258a22887f5bfc4c495fe6ee9f4c328f02b521db29ec495c51821"} Jul 14 19:52:59.917309 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.917162 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbf959699-lg2wp" event={"ID":"e70ccdd9-a7a2-441d-931f-38c86f0ad341","Type":"ContainerStarted","Data":"a97e90ae9cac761c3953515eae104820148375820697e7ed373dbf4b84cd2daf"} Jul 14 19:52:59.935765 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:52:59.935706 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fbf959699-lg2wp" podStartSLOduration=0.935689355 podStartE2EDuration="935.689355ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:59.934115949 +0000 UTC m=+398.013861743" watchObservedRunningTime="2026-07-14 19:52:59.935689355 +0000 UTC m=+398.015435147" Jul 14 19:53:09.418370 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:09.418330 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:53:09.418738 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:09.418383 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:53:09.422921 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:09.422898 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:53:09.961107 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:09.961078 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fbf959699-lg2wp" Jul 14 19:53:26.590999 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:26.590961 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x"] Jul 14 19:53:26.594385 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:26.594363 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" Jul 14 19:53:26.596941 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:26.596915 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 19:53:26.597658 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:26.597633 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mgsdx\"" Jul 14 19:53:26.597767 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:26.597633 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 19:53:26.605544 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:26.605519 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x"] Jul 14 19:53:26.632434 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:26.632396 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzfp5\" (UniqueName: \"kubernetes.io/projected/0f50d7dd-ee0f-46b8-9119-3f4c4790aac3-kube-api-access-hzfp5\") pod \"limitador-operator-controller-manager-859d7b78bd-4s54x\" (UID: \"0f50d7dd-ee0f-46b8-9119-3f4c4790aac3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" Jul 14 19:53:26.733093 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:26.733038 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzfp5\" (UniqueName: \"kubernetes.io/projected/0f50d7dd-ee0f-46b8-9119-3f4c4790aac3-kube-api-access-hzfp5\") pod \"limitador-operator-controller-manager-859d7b78bd-4s54x\" (UID: \"0f50d7dd-ee0f-46b8-9119-3f4c4790aac3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" Jul 14 19:53:26.741784 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:26.741760 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzfp5\" (UniqueName: \"kubernetes.io/projected/0f50d7dd-ee0f-46b8-9119-3f4c4790aac3-kube-api-access-hzfp5\") pod \"limitador-operator-controller-manager-859d7b78bd-4s54x\" (UID: \"0f50d7dd-ee0f-46b8-9119-3f4c4790aac3\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" Jul 14 19:53:26.904458 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:26.904368 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" Jul 14 19:53:27.072363 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:27.072259 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x"] Jul 14 19:53:27.074755 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:53:27.074727 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f50d7dd_ee0f_46b8_9119_3f4c4790aac3.slice/crio-9b33ad79f66f1bc7d6bb228067e96d1b4fe778408941334822633671558d0259 WatchSource:0}: Error finding container 9b33ad79f66f1bc7d6bb228067e96d1b4fe778408941334822633671558d0259: Status 404 returned error can't find the container with id 9b33ad79f66f1bc7d6bb228067e96d1b4fe778408941334822633671558d0259 Jul 14 19:53:28.036654 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:28.036612 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" event={"ID":"0f50d7dd-ee0f-46b8-9119-3f4c4790aac3","Type":"ContainerStarted","Data":"9b33ad79f66f1bc7d6bb228067e96d1b4fe778408941334822633671558d0259"} Jul 14 19:53:30.048331 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:30.048294 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" event={"ID":"0f50d7dd-ee0f-46b8-9119-3f4c4790aac3","Type":"ContainerStarted","Data":"6d6f6f9dd65d1a73429014e00ab7d497bb6b47180371a524809367f7f24acbd6"} Jul 14 19:53:30.048840 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:30.048436 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" Jul 14 19:53:30.070527 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:30.070473 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" podStartSLOduration=2.135944021 podStartE2EDuration="4.070456096s" totalImagesPullingTime="1.934512075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:53:30.06898848 +0000 UTC m=+428.148734273" watchObservedRunningTime="2026-07-14 19:53:30.070456096 +0000 UTC m=+428.150201941" Jul 14 19:53:41.054151 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:41.054119 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" Jul 14 19:53:41.729571 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:41.729540 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x"] Jul 14 19:53:41.729763 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:41.729739 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" podUID="0f50d7dd-ee0f-46b8-9119-3f4c4790aac3" containerName="manager" containerID="cri-o://6d6f6f9dd65d1a73429014e00ab7d497bb6b47180371a524809367f7f24acbd6" gracePeriod=2 Jul 14 19:53:41.735336 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:53:41.735304 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4s54x\" is forbidden: User \"system:node:ip-10-0-132-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-184.ec2.internal' and this object" podUID="0f50d7dd-ee0f-46b8-9119-3f4c4790aac3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" Jul 14 19:53:41.735526 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:41.735503 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x"] Jul 14 19:53:41.796595 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:41.796566 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-74x5n"] Jul 14 19:53:41.796870 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:41.796859 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f50d7dd-ee0f-46b8-9119-3f4c4790aac3" containerName="manager" Jul 14 19:53:41.796913 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:41.796872 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f50d7dd-ee0f-46b8-9119-3f4c4790aac3" containerName="manager" Jul 14 19:53:41.796946 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:41.796932 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f50d7dd-ee0f-46b8-9119-3f4c4790aac3" containerName="manager" Jul 14 19:53:41.799821 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:41.799801 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-74x5n" Jul 14 19:53:41.829231 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:53:41.829197 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4s54x\" is forbidden: User \"system:node:ip-10-0-132-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-184.ec2.internal' and this object" podUID="0f50d7dd-ee0f-46b8-9119-3f4c4790aac3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" Jul 14 19:53:41.860811 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:41.860771 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2h52\" (UniqueName: \"kubernetes.io/projected/95f534a0-4a84-42c0-ac9b-3f7b354d4a1b-kube-api-access-s2h52\") pod \"limitador-operator-controller-manager-859d7b78bd-74x5n\" (UID: \"95f534a0-4a84-42c0-ac9b-3f7b354d4a1b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-74x5n" Jul 14 19:53:41.861842 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:41.861818 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-74x5n"] Jul 14 19:53:41.961227 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:41.961200 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2h52\" (UniqueName: \"kubernetes.io/projected/95f534a0-4a84-42c0-ac9b-3f7b354d4a1b-kube-api-access-s2h52\") pod \"limitador-operator-controller-manager-859d7b78bd-74x5n\" (UID: \"95f534a0-4a84-42c0-ac9b-3f7b354d4a1b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-74x5n" Jul 14 19:53:41.963881 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:41.963864 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" Jul 14 19:53:41.965638 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:53:41.965614 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4s54x\" is forbidden: User \"system:node:ip-10-0-132-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-184.ec2.internal' and this object" podUID="0f50d7dd-ee0f-46b8-9119-3f4c4790aac3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" Jul 14 19:53:41.973489 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:41.973470 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2h52\" (UniqueName: \"kubernetes.io/projected/95f534a0-4a84-42c0-ac9b-3f7b354d4a1b-kube-api-access-s2h52\") pod \"limitador-operator-controller-manager-859d7b78bd-74x5n\" (UID: \"95f534a0-4a84-42c0-ac9b-3f7b354d4a1b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-74x5n" Jul 14 19:53:42.062252 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:42.062163 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzfp5\" (UniqueName: \"kubernetes.io/projected/0f50d7dd-ee0f-46b8-9119-3f4c4790aac3-kube-api-access-hzfp5\") pod \"0f50d7dd-ee0f-46b8-9119-3f4c4790aac3\" (UID: \"0f50d7dd-ee0f-46b8-9119-3f4c4790aac3\") " Jul 14 19:53:42.064243 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:42.064218 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f50d7dd-ee0f-46b8-9119-3f4c4790aac3-kube-api-access-hzfp5" (OuterVolumeSpecName: "kube-api-access-hzfp5") pod "0f50d7dd-ee0f-46b8-9119-3f4c4790aac3" (UID: "0f50d7dd-ee0f-46b8-9119-3f4c4790aac3"). InnerVolumeSpecName "kube-api-access-hzfp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:53:42.098570 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:42.098539 2557 generic.go:356] "Generic (PLEG): container finished" podID="0f50d7dd-ee0f-46b8-9119-3f4c4790aac3" containerID="6d6f6f9dd65d1a73429014e00ab7d497bb6b47180371a524809367f7f24acbd6" exitCode=0 Jul 14 19:53:42.098718 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:42.098587 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" Jul 14 19:53:42.098718 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:42.098624 2557 scope.go:117] "RemoveContainer" containerID="6d6f6f9dd65d1a73429014e00ab7d497bb6b47180371a524809367f7f24acbd6" Jul 14 19:53:42.100802 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:53:42.100772 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4s54x\" is forbidden: User \"system:node:ip-10-0-132-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-184.ec2.internal' and this object" podUID="0f50d7dd-ee0f-46b8-9119-3f4c4790aac3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" Jul 14 19:53:42.107027 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:42.107009 2557 scope.go:117] "RemoveContainer" containerID="6d6f6f9dd65d1a73429014e00ab7d497bb6b47180371a524809367f7f24acbd6" Jul 14 19:53:42.107332 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:53:42.107313 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d6f6f9dd65d1a73429014e00ab7d497bb6b47180371a524809367f7f24acbd6\": container with ID starting with 6d6f6f9dd65d1a73429014e00ab7d497bb6b47180371a524809367f7f24acbd6 not found: ID does not exist" containerID="6d6f6f9dd65d1a73429014e00ab7d497bb6b47180371a524809367f7f24acbd6" Jul 14 19:53:42.107375 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:42.107342 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d6f6f9dd65d1a73429014e00ab7d497bb6b47180371a524809367f7f24acbd6"} err="failed to get container status \"6d6f6f9dd65d1a73429014e00ab7d497bb6b47180371a524809367f7f24acbd6\": rpc error: code = NotFound desc = could not find container \"6d6f6f9dd65d1a73429014e00ab7d497bb6b47180371a524809367f7f24acbd6\": container with ID starting with 6d6f6f9dd65d1a73429014e00ab7d497bb6b47180371a524809367f7f24acbd6 not found: ID does not exist" Jul 14 19:53:42.108659 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:53:42.108638 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4s54x\" is forbidden: User \"system:node:ip-10-0-132-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-184.ec2.internal' and this object" podUID="0f50d7dd-ee0f-46b8-9119-3f4c4790aac3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" Jul 14 19:53:42.109714 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:42.109702 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-74x5n" Jul 14 19:53:42.163245 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:42.163215 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzfp5\" (UniqueName: \"kubernetes.io/projected/0f50d7dd-ee0f-46b8-9119-3f4c4790aac3-kube-api-access-hzfp5\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:53:42.237133 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:42.237039 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-74x5n"] Jul 14 19:53:42.239364 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:53:42.239341 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f534a0_4a84_42c0_ac9b_3f7b354d4a1b.slice/crio-d77496d4a837792ec32a668d7830186b271fb6c9e6babba959d528d205dd0c7a WatchSource:0}: Error finding container d77496d4a837792ec32a668d7830186b271fb6c9e6babba959d528d205dd0c7a: Status 404 returned error can't find the container with id d77496d4a837792ec32a668d7830186b271fb6c9e6babba959d528d205dd0c7a Jul 14 19:53:42.359342 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:53:42.359300 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-4s54x\" is forbidden: User \"system:node:ip-10-0-132-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-184.ec2.internal' and this object" podUID="0f50d7dd-ee0f-46b8-9119-3f4c4790aac3" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4s54x" Jul 14 19:53:42.360623 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:42.360600 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f50d7dd-ee0f-46b8-9119-3f4c4790aac3" path="/var/lib/kubelet/pods/0f50d7dd-ee0f-46b8-9119-3f4c4790aac3/volumes" Jul 14 19:53:43.105273 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:43.105241 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-74x5n" event={"ID":"95f534a0-4a84-42c0-ac9b-3f7b354d4a1b","Type":"ContainerStarted","Data":"fbfb98bd4f7ab50209d2bad3d16495cf563b272853c0d779bb6991ac3f042874"} Jul 14 19:53:43.105702 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:43.105287 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-74x5n" event={"ID":"95f534a0-4a84-42c0-ac9b-3f7b354d4a1b","Type":"ContainerStarted","Data":"d77496d4a837792ec32a668d7830186b271fb6c9e6babba959d528d205dd0c7a"} Jul 14 19:53:43.105702 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:43.105391 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-74x5n" Jul 14 19:53:43.122128 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:43.122076 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-74x5n" podStartSLOduration=2.122039758 podStartE2EDuration="2.122039758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:53:43.121980109 +0000 UTC m=+441.201725924" watchObservedRunningTime="2026-07-14 19:53:43.122039758 +0000 UTC m=+441.201785549" Jul 14 19:53:54.112127 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:53:54.112043 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-74x5n" Jul 14 19:54:27.222035 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.221995 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 19:54:27.226381 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.226357 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8nkkl" Jul 14 19:54:27.228464 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.228436 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 14 19:54:27.228603 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.228548 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-slfsg\"" Jul 14 19:54:27.234588 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.234561 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 19:54:27.247169 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.247143 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 19:54:27.335376 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.335344 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7bd8be7c-548c-4bc7-9d2d-69cf3457ee82-config-file\") pod \"limitador-limitador-54d6d988d5-8nkkl\" (UID: \"7bd8be7c-548c-4bc7-9d2d-69cf3457ee82\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8nkkl" Jul 14 19:54:27.335376 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.335382 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbv2l\" (UniqueName: \"kubernetes.io/projected/7bd8be7c-548c-4bc7-9d2d-69cf3457ee82-kube-api-access-mbv2l\") pod \"limitador-limitador-54d6d988d5-8nkkl\" (UID: \"7bd8be7c-548c-4bc7-9d2d-69cf3457ee82\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8nkkl" Jul 14 19:54:27.435896 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.435857 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7bd8be7c-548c-4bc7-9d2d-69cf3457ee82-config-file\") pod \"limitador-limitador-54d6d988d5-8nkkl\" (UID: \"7bd8be7c-548c-4bc7-9d2d-69cf3457ee82\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8nkkl" Jul 14 19:54:27.435896 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.435901 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbv2l\" (UniqueName: \"kubernetes.io/projected/7bd8be7c-548c-4bc7-9d2d-69cf3457ee82-kube-api-access-mbv2l\") pod \"limitador-limitador-54d6d988d5-8nkkl\" (UID: \"7bd8be7c-548c-4bc7-9d2d-69cf3457ee82\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8nkkl" Jul 14 19:54:27.436569 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.436545 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7bd8be7c-548c-4bc7-9d2d-69cf3457ee82-config-file\") pod \"limitador-limitador-54d6d988d5-8nkkl\" (UID: \"7bd8be7c-548c-4bc7-9d2d-69cf3457ee82\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8nkkl" Jul 14 19:54:27.444429 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.444399 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbv2l\" (UniqueName: \"kubernetes.io/projected/7bd8be7c-548c-4bc7-9d2d-69cf3457ee82-kube-api-access-mbv2l\") pod \"limitador-limitador-54d6d988d5-8nkkl\" (UID: \"7bd8be7c-548c-4bc7-9d2d-69cf3457ee82\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8nkkl" Jul 14 19:54:27.536900 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.536817 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8nkkl" Jul 14 19:54:27.665331 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.665254 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 19:54:27.668034 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:54:27.668005 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd8be7c_548c_4bc7_9d2d_69cf3457ee82.slice/crio-70b571c93f72625e356106f2878b55b75e26f0545d16135733b6e8e70253a476 WatchSource:0}: Error finding container 70b571c93f72625e356106f2878b55b75e26f0545d16135733b6e8e70253a476: Status 404 returned error can't find the container with id 70b571c93f72625e356106f2878b55b75e26f0545d16135733b6e8e70253a476 Jul 14 19:54:27.933606 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.933566 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-7zkgw"] Jul 14 19:54:27.938046 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.938029 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7zkgw" Jul 14 19:54:27.940268 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.940234 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5q7rj\"" Jul 14 19:54:27.941704 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:27.941684 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7zkgw"] Jul 14 19:54:28.040794 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:28.040759 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2x2q\" (UniqueName: \"kubernetes.io/projected/1693a6a3-37c7-4673-ad87-372ebd34d0d7-kube-api-access-x2x2q\") pod \"authorino-75d556bd77-7zkgw\" (UID: \"1693a6a3-37c7-4673-ad87-372ebd34d0d7\") " pod="kuadrant-system/authorino-75d556bd77-7zkgw" Jul 14 19:54:28.142228 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:28.142187 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2x2q\" (UniqueName: \"kubernetes.io/projected/1693a6a3-37c7-4673-ad87-372ebd34d0d7-kube-api-access-x2x2q\") pod \"authorino-75d556bd77-7zkgw\" (UID: \"1693a6a3-37c7-4673-ad87-372ebd34d0d7\") " pod="kuadrant-system/authorino-75d556bd77-7zkgw" Jul 14 19:54:28.154577 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:28.154547 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2x2q\" (UniqueName: \"kubernetes.io/projected/1693a6a3-37c7-4673-ad87-372ebd34d0d7-kube-api-access-x2x2q\") pod \"authorino-75d556bd77-7zkgw\" (UID: \"1693a6a3-37c7-4673-ad87-372ebd34d0d7\") " pod="kuadrant-system/authorino-75d556bd77-7zkgw" Jul 14 19:54:28.247854 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:28.247767 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7zkgw" Jul 14 19:54:28.300528 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:28.300471 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8nkkl" event={"ID":"7bd8be7c-548c-4bc7-9d2d-69cf3457ee82","Type":"ContainerStarted","Data":"70b571c93f72625e356106f2878b55b75e26f0545d16135733b6e8e70253a476"} Jul 14 19:54:28.412308 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:28.412278 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7zkgw"] Jul 14 19:54:28.415045 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:54:28.415013 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1693a6a3_37c7_4673_ad87_372ebd34d0d7.slice/crio-7aa2cdce6b274496dbf0f8db3fd74787d3f481d4a2ef5bd6e9c25fc3a5055395 WatchSource:0}: Error finding container 7aa2cdce6b274496dbf0f8db3fd74787d3f481d4a2ef5bd6e9c25fc3a5055395: Status 404 returned error can't find the container with id 7aa2cdce6b274496dbf0f8db3fd74787d3f481d4a2ef5bd6e9c25fc3a5055395 Jul 14 19:54:29.307932 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:29.307884 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7zkgw" event={"ID":"1693a6a3-37c7-4673-ad87-372ebd34d0d7","Type":"ContainerStarted","Data":"7aa2cdce6b274496dbf0f8db3fd74787d3f481d4a2ef5bd6e9c25fc3a5055395"} Jul 14 19:54:32.323277 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:32.323184 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7zkgw" event={"ID":"1693a6a3-37c7-4673-ad87-372ebd34d0d7","Type":"ContainerStarted","Data":"9341b4e674256ab8ccbd3b3ae159320c8b6bea4b7a45b0cd44becd0c6125dd8a"} Jul 14 19:54:32.324562 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:32.324539 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8nkkl" event={"ID":"7bd8be7c-548c-4bc7-9d2d-69cf3457ee82","Type":"ContainerStarted","Data":"37bf9b50c5b9cc16d1e348539d3d42144e2dba277a8ca770c572aceb5cdc849b"} Jul 14 19:54:32.324682 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:32.324669 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8nkkl" Jul 14 19:54:32.341252 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:32.341207 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-7zkgw" podStartSLOduration=1.698092982 podStartE2EDuration="5.341193161s" totalImagesPullingTime="3.643100179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:54:32.340819309 +0000 UTC m=+490.420565105" watchObservedRunningTime="2026-07-14 19:54:32.341193161 +0000 UTC m=+490.420938955" Jul 14 19:54:32.357692 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:32.357642 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-8nkkl" podStartSLOduration=1.022305234 podStartE2EDuration="5.357623846s" totalImagesPullingTime="4.335318612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:54:32.35647901 +0000 UTC m=+490.436224804" watchObservedRunningTime="2026-07-14 19:54:32.357623846 +0000 UTC m=+490.437369640" Jul 14 19:54:43.330239 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:54:43.330209 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8nkkl" Jul 14 19:55:02.199530 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:02.199498 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68dbbd5565-9tvqt"] Jul 14 19:55:02.206206 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:02.206187 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68dbbd5565-9tvqt" Jul 14 19:55:02.209947 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:02.209923 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68dbbd5565-9tvqt"] Jul 14 19:55:02.317038 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:02.316998 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68dbbd5565-9tvqt"] Jul 14 19:55:02.317316 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:55:02.317294 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-zxkh4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-68dbbd5565-9tvqt" podUID="be92df8c-ae5d-4641-a6c4-bb996b29487f" Jul 14 19:55:02.350079 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:02.350021 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxkh4\" (UniqueName: \"kubernetes.io/projected/be92df8c-ae5d-4641-a6c4-bb996b29487f-kube-api-access-zxkh4\") pod \"authorino-68dbbd5565-9tvqt\" (UID: \"be92df8c-ae5d-4641-a6c4-bb996b29487f\") " pod="kuadrant-system/authorino-68dbbd5565-9tvqt" Jul 14 19:55:02.450632 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:02.450553 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxkh4\" (UniqueName: \"kubernetes.io/projected/be92df8c-ae5d-4641-a6c4-bb996b29487f-kube-api-access-zxkh4\") pod \"authorino-68dbbd5565-9tvqt\" (UID: \"be92df8c-ae5d-4641-a6c4-bb996b29487f\") " pod="kuadrant-system/authorino-68dbbd5565-9tvqt" Jul 14 19:55:02.459222 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:02.459198 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxkh4\" (UniqueName: \"kubernetes.io/projected/be92df8c-ae5d-4641-a6c4-bb996b29487f-kube-api-access-zxkh4\") pod \"authorino-68dbbd5565-9tvqt\" (UID: \"be92df8c-ae5d-4641-a6c4-bb996b29487f\") " pod="kuadrant-system/authorino-68dbbd5565-9tvqt" Jul 14 19:55:02.464825 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:02.464799 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68dbbd5565-9tvqt" Jul 14 19:55:02.471784 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:02.471764 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68dbbd5565-9tvqt" Jul 14 19:55:02.551240 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:02.551208 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxkh4\" (UniqueName: \"kubernetes.io/projected/be92df8c-ae5d-4641-a6c4-bb996b29487f-kube-api-access-zxkh4\") pod \"be92df8c-ae5d-4641-a6c4-bb996b29487f\" (UID: \"be92df8c-ae5d-4641-a6c4-bb996b29487f\") " Jul 14 19:55:02.553459 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:02.553429 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be92df8c-ae5d-4641-a6c4-bb996b29487f-kube-api-access-zxkh4" (OuterVolumeSpecName: "kube-api-access-zxkh4") pod "be92df8c-ae5d-4641-a6c4-bb996b29487f" (UID: "be92df8c-ae5d-4641-a6c4-bb996b29487f"). InnerVolumeSpecName "kube-api-access-zxkh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:55:02.652155 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:02.652117 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxkh4\" (UniqueName: \"kubernetes.io/projected/be92df8c-ae5d-4641-a6c4-bb996b29487f-kube-api-access-zxkh4\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:55:03.433228 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:03.433187 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7zkgw"] Jul 14 19:55:03.433697 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:03.433422 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-7zkgw" podUID="1693a6a3-37c7-4673-ad87-372ebd34d0d7" containerName="authorino" containerID="cri-o://9341b4e674256ab8ccbd3b3ae159320c8b6bea4b7a45b0cd44becd0c6125dd8a" gracePeriod=30 Jul 14 19:55:03.469341 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:03.469312 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68dbbd5565-9tvqt" Jul 14 19:55:03.517902 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:03.517872 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68dbbd5565-9tvqt"] Jul 14 19:55:03.523168 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:03.523142 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-68dbbd5565-9tvqt"] Jul 14 19:55:03.688477 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:03.688425 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7zkgw" Jul 14 19:55:03.759402 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:03.759368 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2x2q\" (UniqueName: \"kubernetes.io/projected/1693a6a3-37c7-4673-ad87-372ebd34d0d7-kube-api-access-x2x2q\") pod \"1693a6a3-37c7-4673-ad87-372ebd34d0d7\" (UID: \"1693a6a3-37c7-4673-ad87-372ebd34d0d7\") " Jul 14 19:55:03.761418 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:03.761391 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1693a6a3-37c7-4673-ad87-372ebd34d0d7-kube-api-access-x2x2q" (OuterVolumeSpecName: "kube-api-access-x2x2q") pod "1693a6a3-37c7-4673-ad87-372ebd34d0d7" (UID: "1693a6a3-37c7-4673-ad87-372ebd34d0d7"). InnerVolumeSpecName "kube-api-access-x2x2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:55:03.859928 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:03.859892 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2x2q\" (UniqueName: \"kubernetes.io/projected/1693a6a3-37c7-4673-ad87-372ebd34d0d7-kube-api-access-x2x2q\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 19:55:04.360212 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:04.360175 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be92df8c-ae5d-4641-a6c4-bb996b29487f" path="/var/lib/kubelet/pods/be92df8c-ae5d-4641-a6c4-bb996b29487f/volumes" Jul 14 19:55:04.474984 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:04.474946 2557 generic.go:356] "Generic (PLEG): container finished" podID="1693a6a3-37c7-4673-ad87-372ebd34d0d7" containerID="9341b4e674256ab8ccbd3b3ae159320c8b6bea4b7a45b0cd44becd0c6125dd8a" exitCode=0 Jul 14 19:55:04.474984 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:04.474986 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7zkgw" Jul 14 19:55:04.475504 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:04.474992 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7zkgw" event={"ID":"1693a6a3-37c7-4673-ad87-372ebd34d0d7","Type":"ContainerDied","Data":"9341b4e674256ab8ccbd3b3ae159320c8b6bea4b7a45b0cd44becd0c6125dd8a"} Jul 14 19:55:04.475504 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:04.475029 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7zkgw" event={"ID":"1693a6a3-37c7-4673-ad87-372ebd34d0d7","Type":"ContainerDied","Data":"7aa2cdce6b274496dbf0f8db3fd74787d3f481d4a2ef5bd6e9c25fc3a5055395"} Jul 14 19:55:04.475504 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:04.475051 2557 scope.go:117] "RemoveContainer" containerID="9341b4e674256ab8ccbd3b3ae159320c8b6bea4b7a45b0cd44becd0c6125dd8a" Jul 14 19:55:04.483507 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:04.483488 2557 scope.go:117] "RemoveContainer" containerID="9341b4e674256ab8ccbd3b3ae159320c8b6bea4b7a45b0cd44becd0c6125dd8a" Jul 14 19:55:04.483781 ip-10-0-132-184 kubenswrapper[2557]: E0714 19:55:04.483760 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9341b4e674256ab8ccbd3b3ae159320c8b6bea4b7a45b0cd44becd0c6125dd8a\": container with ID starting with 9341b4e674256ab8ccbd3b3ae159320c8b6bea4b7a45b0cd44becd0c6125dd8a not found: ID does not exist" containerID="9341b4e674256ab8ccbd3b3ae159320c8b6bea4b7a45b0cd44becd0c6125dd8a" Jul 14 19:55:04.483838 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:04.483789 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9341b4e674256ab8ccbd3b3ae159320c8b6bea4b7a45b0cd44becd0c6125dd8a"} err="failed to get container status \"9341b4e674256ab8ccbd3b3ae159320c8b6bea4b7a45b0cd44becd0c6125dd8a\": rpc error: code = NotFound desc = could not find container \"9341b4e674256ab8ccbd3b3ae159320c8b6bea4b7a45b0cd44becd0c6125dd8a\": container with ID starting with 9341b4e674256ab8ccbd3b3ae159320c8b6bea4b7a45b0cd44becd0c6125dd8a not found: ID does not exist" Jul 14 19:55:04.492044 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:04.492023 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7zkgw"] Jul 14 19:55:04.495729 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:04.495704 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7zkgw"] Jul 14 19:55:06.359993 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:06.359959 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1693a6a3-37c7-4673-ad87-372ebd34d0d7" path="/var/lib/kubelet/pods/1693a6a3-37c7-4673-ad87-372ebd34d0d7/volumes" Jul 14 19:55:43.895815 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:55:43.895778 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 19:56:18.501804 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:18.501763 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 19:56:22.289843 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:22.289816 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 19:56:22.290729 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:22.290707 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 19:56:26.205421 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:26.205387 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 19:56:31.396887 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:31.396844 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 19:56:36.489722 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.489687 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v"] Jul 14 19:56:36.490188 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.490170 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1693a6a3-37c7-4673-ad87-372ebd34d0d7" containerName="authorino" Jul 14 19:56:36.490188 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.490188 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1693a6a3-37c7-4673-ad87-372ebd34d0d7" containerName="authorino" Jul 14 19:56:36.490304 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.490286 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1693a6a3-37c7-4673-ad87-372ebd34d0d7" containerName="authorino" Jul 14 19:56:36.493576 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.493552 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.496908 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.496883 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 19:56:36.497036 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.496885 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 14 19:56:36.497036 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.496923 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wjq24\"" Jul 14 19:56:36.497036 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.496920 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 19:56:36.504083 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.504047 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v"] Jul 14 19:56:36.648388 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.648352 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63bda19c-fc1e-4c20-9f5e-6c0234759068-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.648568 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.648396 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63bda19c-fc1e-4c20-9f5e-6c0234759068-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.648568 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.648422 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63bda19c-fc1e-4c20-9f5e-6c0234759068-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.648568 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.648507 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63bda19c-fc1e-4c20-9f5e-6c0234759068-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.648568 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.648536 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fnvm\" (UniqueName: \"kubernetes.io/projected/63bda19c-fc1e-4c20-9f5e-6c0234759068-kube-api-access-9fnvm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.648568 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.648559 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63bda19c-fc1e-4c20-9f5e-6c0234759068-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.749123 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.749007 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63bda19c-fc1e-4c20-9f5e-6c0234759068-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.749123 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.749099 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63bda19c-fc1e-4c20-9f5e-6c0234759068-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.749335 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.749138 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63bda19c-fc1e-4c20-9f5e-6c0234759068-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.749335 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.749174 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63bda19c-fc1e-4c20-9f5e-6c0234759068-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.749335 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.749197 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fnvm\" (UniqueName: \"kubernetes.io/projected/63bda19c-fc1e-4c20-9f5e-6c0234759068-kube-api-access-9fnvm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.749335 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.749223 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63bda19c-fc1e-4c20-9f5e-6c0234759068-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.749533 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.749496 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63bda19c-fc1e-4c20-9f5e-6c0234759068-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.749610 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.749561 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63bda19c-fc1e-4c20-9f5e-6c0234759068-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.749610 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.749579 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63bda19c-fc1e-4c20-9f5e-6c0234759068-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.751467 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.751445 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63bda19c-fc1e-4c20-9f5e-6c0234759068-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.751572 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.751538 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63bda19c-fc1e-4c20-9f5e-6c0234759068-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.758420 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.758397 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fnvm\" (UniqueName: \"kubernetes.io/projected/63bda19c-fc1e-4c20-9f5e-6c0234759068-kube-api-access-9fnvm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v\" (UID: \"63bda19c-fc1e-4c20-9f5e-6c0234759068\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.804240 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.804208 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:36.937406 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.937372 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v"] Jul 14 19:56:36.939676 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:56:36.939635 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bda19c_fc1e_4c20_9f5e_6c0234759068.slice/crio-a8c2fa7751944b66c43e5f96a1049936db1b054afa121dcc084bcda487d6c44e WatchSource:0}: Error finding container a8c2fa7751944b66c43e5f96a1049936db1b054afa121dcc084bcda487d6c44e: Status 404 returned error can't find the container with id a8c2fa7751944b66c43e5f96a1049936db1b054afa121dcc084bcda487d6c44e Jul 14 19:56:36.941157 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:36.941138 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:56:37.864943 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:37.864899 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" event={"ID":"63bda19c-fc1e-4c20-9f5e-6c0234759068","Type":"ContainerStarted","Data":"a8c2fa7751944b66c43e5f96a1049936db1b054afa121dcc084bcda487d6c44e"} Jul 14 19:56:38.290783 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:38.290759 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 19:56:38.870249 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:38.870210 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" event={"ID":"63bda19c-fc1e-4c20-9f5e-6c0234759068","Type":"ContainerStarted","Data":"f91dcd7b22df6e88226df2c44f5e73275ee5616ca07124249a0b94cf51ee17fe"} Jul 14 19:56:38.891420 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:38.891371 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" podStartSLOduration=1.542772859 podStartE2EDuration="2.891354762s" totalImagesPullingTime="1.348581903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:56:38.890661777 +0000 UTC m=+616.970407580" watchObservedRunningTime="2026-07-14 19:56:38.891354762 +0000 UTC m=+616.971100553" Jul 14 19:56:46.804793 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:46.804656 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:46.805202 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:46.804821 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:46.817422 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:46.817394 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:46.915771 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:46.915741 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v" Jul 14 19:56:56.088139 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:56:56.088098 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 19:57:04.094349 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.094312 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf"] Jul 14 19:57:04.100801 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.100778 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.102983 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.102951 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 14 19:57:04.109340 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.109317 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf"] Jul 14 19:57:04.178402 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.178370 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.178570 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.178413 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.178570 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.178460 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.178570 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.178509 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.178570 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.178528 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9s7f\" (UniqueName: \"kubernetes.io/projected/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-kube-api-access-t9s7f\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.178570 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.178547 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.279559 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.279520 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.279733 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.279580 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.279733 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.279627 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.279733 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.279674 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.279733 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.279730 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.280051 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.279759 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9s7f\" (UniqueName: \"kubernetes.io/projected/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-kube-api-access-t9s7f\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.280051 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.279990 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.280051 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.280008 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.280229 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.280087 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.281938 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.281916 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.282136 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.282119 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.290011 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.289990 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9s7f\" (UniqueName: \"kubernetes.io/projected/ce6d5f17-7fc2-4535-988f-5dd5fcb74c41-kube-api-access-t9s7f\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf\" (UID: \"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.410897 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.410814 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:04.541626 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.541602 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf"] Jul 14 19:57:04.543918 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:57:04.543897 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce6d5f17_7fc2_4535_988f_5dd5fcb74c41.slice/crio-5670f2f5ce554912d5127e069bde34c6aab09dec1d7e4a4ee9679fe16154686b WatchSource:0}: Error finding container 5670f2f5ce554912d5127e069bde34c6aab09dec1d7e4a4ee9679fe16154686b: Status 404 returned error can't find the container with id 5670f2f5ce554912d5127e069bde34c6aab09dec1d7e4a4ee9679fe16154686b Jul 14 19:57:04.982693 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.982659 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" event={"ID":"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41","Type":"ContainerStarted","Data":"2eb5004b7b5ab192f5e11b3710ceaeb5b816190a9a305528ff9eefc9fbc04edd"} Jul 14 19:57:04.982857 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:04.982706 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" event={"ID":"ce6d5f17-7fc2-4535-988f-5dd5fcb74c41","Type":"ContainerStarted","Data":"5670f2f5ce554912d5127e069bde34c6aab09dec1d7e4a4ee9679fe16154686b"} Jul 14 19:57:05.000614 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:05.000566 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" podStartSLOduration=0.715892226 podStartE2EDuration="1.000549947s" totalImagesPullingTime="284.657721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:57:05.000370901 +0000 UTC m=+643.080116742" watchObservedRunningTime="2026-07-14 19:57:05.000549947 +0000 UTC m=+643.080295751" Jul 14 19:57:08.196301 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:08.196271 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 19:57:14.411905 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:14.411866 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:14.412306 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:14.411928 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:14.424605 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:14.424580 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:57:15.035970 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:57:15.035939 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf" Jul 14 19:58:24.926022 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:24.925933 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76b84bf67b-4qxqp"] Jul 14 19:58:24.929367 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:24.929344 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76b84bf67b-4qxqp" Jul 14 19:58:24.932254 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:24.932234 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 14 19:58:24.932412 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:24.932270 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5q7rj\"" Jul 14 19:58:24.937864 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:24.937840 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76b84bf67b-4qxqp"] Jul 14 19:58:25.003033 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:25.002991 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bszg2\" (UniqueName: \"kubernetes.io/projected/c455ffb7-c1b9-412c-bd91-4c104eef9b5c-kube-api-access-bszg2\") pod \"authorino-76b84bf67b-4qxqp\" (UID: \"c455ffb7-c1b9-412c-bd91-4c104eef9b5c\") " pod="kuadrant-system/authorino-76b84bf67b-4qxqp" Jul 14 19:58:25.003218 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:25.003049 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c455ffb7-c1b9-412c-bd91-4c104eef9b5c-tls-cert\") pod \"authorino-76b84bf67b-4qxqp\" (UID: \"c455ffb7-c1b9-412c-bd91-4c104eef9b5c\") " pod="kuadrant-system/authorino-76b84bf67b-4qxqp" Jul 14 19:58:25.104396 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:25.104357 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bszg2\" (UniqueName: \"kubernetes.io/projected/c455ffb7-c1b9-412c-bd91-4c104eef9b5c-kube-api-access-bszg2\") pod \"authorino-76b84bf67b-4qxqp\" (UID: \"c455ffb7-c1b9-412c-bd91-4c104eef9b5c\") " pod="kuadrant-system/authorino-76b84bf67b-4qxqp" Jul 14 19:58:25.104596 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:25.104437 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c455ffb7-c1b9-412c-bd91-4c104eef9b5c-tls-cert\") pod \"authorino-76b84bf67b-4qxqp\" (UID: \"c455ffb7-c1b9-412c-bd91-4c104eef9b5c\") " pod="kuadrant-system/authorino-76b84bf67b-4qxqp" Jul 14 19:58:25.106810 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:25.106783 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c455ffb7-c1b9-412c-bd91-4c104eef9b5c-tls-cert\") pod \"authorino-76b84bf67b-4qxqp\" (UID: \"c455ffb7-c1b9-412c-bd91-4c104eef9b5c\") " pod="kuadrant-system/authorino-76b84bf67b-4qxqp" Jul 14 19:58:25.116440 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:25.116415 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bszg2\" (UniqueName: \"kubernetes.io/projected/c455ffb7-c1b9-412c-bd91-4c104eef9b5c-kube-api-access-bszg2\") pod \"authorino-76b84bf67b-4qxqp\" (UID: \"c455ffb7-c1b9-412c-bd91-4c104eef9b5c\") " pod="kuadrant-system/authorino-76b84bf67b-4qxqp" Jul 14 19:58:25.239574 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:25.239500 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76b84bf67b-4qxqp" Jul 14 19:58:25.369820 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:25.369796 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76b84bf67b-4qxqp"] Jul 14 19:58:25.372131 ip-10-0-132-184 kubenswrapper[2557]: W0714 19:58:25.372103 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc455ffb7_c1b9_412c_bd91_4c104eef9b5c.slice/crio-230d6883c07e65e7fb3ca89235ca4da30a264aebea389a352096826a07a656c9 WatchSource:0}: Error finding container 230d6883c07e65e7fb3ca89235ca4da30a264aebea389a352096826a07a656c9: Status 404 returned error can't find the container with id 230d6883c07e65e7fb3ca89235ca4da30a264aebea389a352096826a07a656c9 Jul 14 19:58:26.340335 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:26.340290 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76b84bf67b-4qxqp" event={"ID":"c455ffb7-c1b9-412c-bd91-4c104eef9b5c","Type":"ContainerStarted","Data":"eb8d05fa5ac297d94b0ddf533d623d6b28353e4d5af6e8715a60fc79e8f22759"} Jul 14 19:58:26.340335 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:26.340334 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76b84bf67b-4qxqp" event={"ID":"c455ffb7-c1b9-412c-bd91-4c104eef9b5c","Type":"ContainerStarted","Data":"230d6883c07e65e7fb3ca89235ca4da30a264aebea389a352096826a07a656c9"} Jul 14 19:58:26.359557 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:26.359494 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-76b84bf67b-4qxqp" podStartSLOduration=1.93953055 podStartE2EDuration="2.359471071s" totalImagesPullingTime="419.940521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:58:26.357187081 +0000 UTC m=+724.436932874" watchObservedRunningTime="2026-07-14 19:58:26.359471071 +0000 UTC m=+724.439216861" Jul 14 19:58:55.995428 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:58:55.995385 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 19:59:07.396869 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:59:07.396831 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 19:59:16.805333 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:59:16.805298 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 19:59:27.586549 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:59:27.586516 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 19:59:37.393659 ip-10-0-132-184 kubenswrapper[2557]: I0714 19:59:37.393626 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:00:03.995748 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:00:03.995706 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:00:17.689341 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:00:17.689310 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:00:49.597614 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:00:49.597578 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:01:00.986833 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:01:00.986790 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:01:10.393730 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:01:10.393693 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:01:19.201901 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:01:19.201826 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:01:22.315913 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:01:22.315881 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 20:01:22.316698 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:01:22.316678 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 20:01:30.199071 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:01:30.199024 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:01:40.199752 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:01:40.199710 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:01:51.000207 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:01:51.000170 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:02:01.193779 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:02:01.193739 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:02:26.495787 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:02:26.495750 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:02:37.197875 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:02:37.197841 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:02:59.894314 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:02:59.894271 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:03:12.494285 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:03:12.494245 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:03:26.799515 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:03:26.799476 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:03:35.991390 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:03:35.991349 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:03:47.197553 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:03:47.197521 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:03:56.503004 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:03:56.502968 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:04:07.695002 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:04:07.694967 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:05:56.325985 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:05:56.325945 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:06:05.891619 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:06:05.891582 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:06:17.298599 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:06:17.298561 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:06:22.342666 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:06:22.342640 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 20:06:22.344231 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:06:22.344207 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 20:06:28.801093 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:06:28.801042 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:06:40.596463 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:06:40.596427 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:06:50.787518 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:06:50.787486 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:07:00.095570 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:07:00.095534 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:07:11.091200 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:07:11.091165 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:07:21.893172 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:07:21.893089 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:07:31.698307 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:07:31.698260 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:07:42.203280 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:07:42.203241 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:08:03.424335 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:08:03.424300 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:08:12.892339 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:08:12.892300 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:08:24.696412 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:08:24.696375 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:08:32.597107 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:08:32.597074 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:08:43.495728 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:08:43.495693 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:09:01.402921 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:09:01.402879 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:09:09.808791 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:09:09.808756 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:09:20.191323 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:09:20.191287 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:09:27.798411 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:09:27.798369 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:09:38.603951 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:09:38.603916 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:09:48.490270 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:09:48.490227 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:10:02.106749 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:10:02.106709 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:10:11.288402 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:10:11.288367 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:10:20.997641 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:10:20.997555 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:10:33.820776 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:10:33.820737 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:10:46.797616 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:10:46.797580 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:10:58.194032 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:10:58.193995 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:11:08.798232 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:11:08.798193 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:11:22.104794 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:11:22.104761 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:11:22.373297 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:11:22.373211 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 20:11:22.376391 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:11:22.376368 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 20:11:34.996650 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:11:34.996617 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:11:48.439506 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:11:48.439422 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:11:56.993294 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:11:56.993258 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:12:06.896051 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:12:06.896011 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:12:18.285653 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:12:18.285618 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:12:26.591782 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:12:26.591743 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:12:48.590357 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:12:48.590319 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:13:48.394201 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:13:48.394162 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:14:02.390078 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:14:02.390025 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:15:17.994780 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:15:17.994745 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:15:34.402867 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:15:34.402824 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:16:22.404438 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:16:22.404406 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 20:16:22.408302 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:16:22.408281 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 20:16:40.190898 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:16:40.190860 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:18:16.699255 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:18:16.699221 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:18:24.994749 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:18:24.994714 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:18:37.694658 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:18:37.694618 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:18:58.295383 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:18:58.295350 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:19:40.608986 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:40.608948 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:19:52.312310 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.312269 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6"] Jul 14 20:19:52.315924 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.315900 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.319327 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.319303 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4d0ddec0\"/\"kube-root-ca.crt\"" Jul 14 20:19:52.319327 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.319312 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4d0ddec0\"/\"e2e-lifecycle-model-4d0ddec0-kserve-self-signed-certs\"" Jul 14 20:19:52.319500 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.319303 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4d0ddec0\"/\"default-dockercfg-mcrvc\"" Jul 14 20:19:52.319500 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.319303 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4d0ddec0\"/\"openshift-service-ca.crt\"" Jul 14 20:19:52.324989 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.324958 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6"] Jul 14 20:19:52.329331 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.329310 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e72e8ed-84a3-4459-8a10-8b95e736d20c-tls-certs\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.329431 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.329360 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-dshm\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.329431 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.329384 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-home\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.329431 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.329421 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gflnw\" (UniqueName: \"kubernetes.io/projected/3e72e8ed-84a3-4459-8a10-8b95e736d20c-kube-api-access-gflnw\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.329545 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.329447 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-model-cache\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.329545 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.329486 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-tmp-dir\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.429995 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.429960 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gflnw\" (UniqueName: \"kubernetes.io/projected/3e72e8ed-84a3-4459-8a10-8b95e736d20c-kube-api-access-gflnw\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.429995 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.430003 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-model-cache\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.430254 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.430041 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-tmp-dir\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.430254 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.430113 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e72e8ed-84a3-4459-8a10-8b95e736d20c-tls-certs\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.430254 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.430165 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-dshm\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.430254 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.430197 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-home\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.430474 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.430450 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-tmp-dir\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.430533 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.430510 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-model-cache\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.430586 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.430537 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-home\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.432337 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.432313 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-dshm\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.432652 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.432636 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e72e8ed-84a3-4459-8a10-8b95e736d20c-tls-certs\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.438426 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.438403 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gflnw\" (UniqueName: \"kubernetes.io/projected/3e72e8ed-84a3-4459-8a10-8b95e736d20c-kube-api-access-gflnw\") pod \"e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.625208 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.625175 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:19:52.759622 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:52.759598 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6"] Jul 14 20:19:52.761988 ip-10-0-132-184 kubenswrapper[2557]: W0714 20:19:52.761956 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e72e8ed_84a3_4459_8a10_8b95e736d20c.slice/crio-dde6c6eef5f69928e2fe6de47d513e502ac2130745680db2a30c3024503506ec WatchSource:0}: Error finding container dde6c6eef5f69928e2fe6de47d513e502ac2130745680db2a30c3024503506ec: Status 404 returned error can't find the container with id dde6c6eef5f69928e2fe6de47d513e502ac2130745680db2a30c3024503506ec Jul 14 20:19:53.009403 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:53.009312 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" event={"ID":"3e72e8ed-84a3-4459-8a10-8b95e736d20c","Type":"ContainerStarted","Data":"cc098cde43d1dbfbad934ed71ab31e2cab17a3dbd8a046205748162601bd0c96"} Jul 14 20:19:53.009403 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:53.009353 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" event={"ID":"3e72e8ed-84a3-4459-8a10-8b95e736d20c","Type":"ContainerStarted","Data":"dde6c6eef5f69928e2fe6de47d513e502ac2130745680db2a30c3024503506ec"} Jul 14 20:19:53.031269 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:53.031208 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" podStartSLOduration=1.031188996 podStartE2EDuration="1.031188996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:19:53.0293774 +0000 UTC m=+2011.109123210" watchObservedRunningTime="2026-07-14 20:19:53.031188996 +0000 UTC m=+2011.110934788" Jul 14 20:19:55.302197 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:19:55.302158 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:20:02.626082 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:02.626024 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:20:02.626562 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:02.626103 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:20:02.639110 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:02.639079 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:20:03.062843 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:03.062758 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:20:13.692666 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:13.692629 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:20:30.891868 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:30.891829 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:20:36.832813 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:36.832781 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6"] Jul 14 20:20:51.852984 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:51.852879 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" podUID="3e72e8ed-84a3-4459-8a10-8b95e736d20c" containerName="main" containerID="cri-o://cc098cde43d1dbfbad934ed71ab31e2cab17a3dbd8a046205748162601bd0c96" gracePeriod=45 Jul 14 20:20:52.094255 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.094223 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:20:52.152839 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.152743 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-model-cache\") pod \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " Jul 14 20:20:52.152839 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.152817 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gflnw\" (UniqueName: \"kubernetes.io/projected/3e72e8ed-84a3-4459-8a10-8b95e736d20c-kube-api-access-gflnw\") pod \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " Jul 14 20:20:52.153024 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.152852 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-tmp-dir\") pod \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " Jul 14 20:20:52.153024 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.152889 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e72e8ed-84a3-4459-8a10-8b95e736d20c-tls-certs\") pod \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " Jul 14 20:20:52.153024 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.152971 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-dshm\") pod \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " Jul 14 20:20:52.153024 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.152997 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-home\") pod \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\" (UID: \"3e72e8ed-84a3-4459-8a10-8b95e736d20c\") " Jul 14 20:20:52.153024 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.153002 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e72e8ed-84a3-4459-8a10-8b95e736d20c" (UID: "3e72e8ed-84a3-4459-8a10-8b95e736d20c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:20:52.153323 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.153160 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e72e8ed-84a3-4459-8a10-8b95e736d20c" (UID: "3e72e8ed-84a3-4459-8a10-8b95e736d20c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:20:52.153380 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.153314 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-home" (OuterVolumeSpecName: "home") pod "3e72e8ed-84a3-4459-8a10-8b95e736d20c" (UID: "3e72e8ed-84a3-4459-8a10-8b95e736d20c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:20:52.153380 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.153326 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-model-cache\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:20:52.153380 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.153345 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-tmp-dir\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:20:52.155215 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.155190 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-dshm" (OuterVolumeSpecName: "dshm") pod "3e72e8ed-84a3-4459-8a10-8b95e736d20c" (UID: "3e72e8ed-84a3-4459-8a10-8b95e736d20c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:20:52.155215 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.155201 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e72e8ed-84a3-4459-8a10-8b95e736d20c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e72e8ed-84a3-4459-8a10-8b95e736d20c" (UID: "3e72e8ed-84a3-4459-8a10-8b95e736d20c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:20:52.155366 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.155212 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e72e8ed-84a3-4459-8a10-8b95e736d20c-kube-api-access-gflnw" (OuterVolumeSpecName: "kube-api-access-gflnw") pod "3e72e8ed-84a3-4459-8a10-8b95e736d20c" (UID: "3e72e8ed-84a3-4459-8a10-8b95e736d20c"). InnerVolumeSpecName "kube-api-access-gflnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:20:52.253973 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.253937 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gflnw\" (UniqueName: \"kubernetes.io/projected/3e72e8ed-84a3-4459-8a10-8b95e736d20c-kube-api-access-gflnw\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:20:52.253973 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.253967 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e72e8ed-84a3-4459-8a10-8b95e736d20c-tls-certs\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:20:52.253973 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.253979 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-dshm\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:20:52.254235 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.253989 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e72e8ed-84a3-4459-8a10-8b95e736d20c-home\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:20:52.276425 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.276388 2557 generic.go:356] "Generic (PLEG): container finished" podID="3e72e8ed-84a3-4459-8a10-8b95e736d20c" containerID="cc098cde43d1dbfbad934ed71ab31e2cab17a3dbd8a046205748162601bd0c96" exitCode=0 Jul 14 20:20:52.276574 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.276440 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" Jul 14 20:20:52.276574 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.276465 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" event={"ID":"3e72e8ed-84a3-4459-8a10-8b95e736d20c","Type":"ContainerDied","Data":"cc098cde43d1dbfbad934ed71ab31e2cab17a3dbd8a046205748162601bd0c96"} Jul 14 20:20:52.276574 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.276504 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6" event={"ID":"3e72e8ed-84a3-4459-8a10-8b95e736d20c","Type":"ContainerDied","Data":"dde6c6eef5f69928e2fe6de47d513e502ac2130745680db2a30c3024503506ec"} Jul 14 20:20:52.276574 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.276526 2557 scope.go:117] "RemoveContainer" containerID="cc098cde43d1dbfbad934ed71ab31e2cab17a3dbd8a046205748162601bd0c96" Jul 14 20:20:52.285413 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.285395 2557 scope.go:117] "RemoveContainer" containerID="cc098cde43d1dbfbad934ed71ab31e2cab17a3dbd8a046205748162601bd0c96" Jul 14 20:20:52.285652 ip-10-0-132-184 kubenswrapper[2557]: E0714 20:20:52.285632 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc098cde43d1dbfbad934ed71ab31e2cab17a3dbd8a046205748162601bd0c96\": container with ID starting with cc098cde43d1dbfbad934ed71ab31e2cab17a3dbd8a046205748162601bd0c96 not found: ID does not exist" containerID="cc098cde43d1dbfbad934ed71ab31e2cab17a3dbd8a046205748162601bd0c96" Jul 14 20:20:52.285696 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.285669 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc098cde43d1dbfbad934ed71ab31e2cab17a3dbd8a046205748162601bd0c96"} err="failed to get container status \"cc098cde43d1dbfbad934ed71ab31e2cab17a3dbd8a046205748162601bd0c96\": rpc error: code = NotFound desc = could not find container \"cc098cde43d1dbfbad934ed71ab31e2cab17a3dbd8a046205748162601bd0c96\": container with ID starting with cc098cde43d1dbfbad934ed71ab31e2cab17a3dbd8a046205748162601bd0c96 not found: ID does not exist" Jul 14 20:20:52.299677 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.299648 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6"] Jul 14 20:20:52.303968 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.303946 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-4d0ddec0/e2e-lifecycle-model-4d0ddec0-kserve-5967c57995-nj9c6"] Jul 14 20:20:52.361831 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:20:52.361800 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e72e8ed-84a3-4459-8a10-8b95e736d20c" path="/var/lib/kubelet/pods/3e72e8ed-84a3-4459-8a10-8b95e736d20c/volumes" Jul 14 20:21:22.430098 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:21:22.430048 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 20:21:22.434264 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:21:22.434243 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 20:21:48.103865 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:21:48.103831 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:22:01.990700 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:22:01.990662 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:22:49.491266 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:22:49.491234 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:23:00.792990 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:00.792959 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:23:04.592647 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.592615 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst"] Jul 14 20:23:04.593122 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.593105 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e72e8ed-84a3-4459-8a10-8b95e736d20c" containerName="main" Jul 14 20:23:04.593198 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.593126 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e72e8ed-84a3-4459-8a10-8b95e736d20c" containerName="main" Jul 14 20:23:04.593258 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.593247 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e72e8ed-84a3-4459-8a10-8b95e736d20c" containerName="main" Jul 14 20:23:04.596137 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.596118 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.599027 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.599003 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1c67fa\"/\"test-model-1c67fa-kserve-self-signed-certs\"" Jul 14 20:23:04.599159 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.599003 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1c67fa\"/\"kube-root-ca.crt\"" Jul 14 20:23:04.599159 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.599002 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1c67fa\"/\"default-dockercfg-gmjdg\"" Jul 14 20:23:04.599159 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.599013 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1c67fa\"/\"openshift-service-ca.crt\"" Jul 14 20:23:04.606271 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.606248 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst"] Jul 14 20:23:04.721706 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.721671 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-home\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.721706 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.721704 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-dshm\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.721952 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.721740 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-model-cache\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.721952 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.721803 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-tmp-dir\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.721952 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.721870 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-tls-certs\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.721952 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.721933 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b49l\" (UniqueName: \"kubernetes.io/projected/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-kube-api-access-9b49l\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.822960 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.822922 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-home\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.822960 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.822961 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-dshm\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.823256 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.822997 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-model-cache\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.823256 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.823022 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-tmp-dir\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.823256 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.823045 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-tls-certs\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.823256 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.823106 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b49l\" (UniqueName: \"kubernetes.io/projected/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-kube-api-access-9b49l\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.823465 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.823418 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-home\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.823465 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.823430 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-model-cache\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.823549 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.823474 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-tmp-dir\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.825345 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.825327 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-dshm\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.825611 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.825595 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-tls-certs\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.831731 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.831707 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b49l\" (UniqueName: \"kubernetes.io/projected/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-kube-api-access-9b49l\") pod \"test-model-1c67fa-kserve-58d47fd677-zvpst\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:04.905644 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:04.905546 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:05.038705 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:05.038551 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst"] Jul 14 20:23:05.041223 ip-10-0-132-184 kubenswrapper[2557]: W0714 20:23:05.041194 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54476a5d_bb21_498c_b449_ad8f4fd2ce0d.slice/crio-52a72fa2bbb61b4143f64a68a492739524046dbd1c8d4a7746743a86e80c7781 WatchSource:0}: Error finding container 52a72fa2bbb61b4143f64a68a492739524046dbd1c8d4a7746743a86e80c7781: Status 404 returned error can't find the container with id 52a72fa2bbb61b4143f64a68a492739524046dbd1c8d4a7746743a86e80c7781 Jul 14 20:23:05.869323 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:05.869287 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" event={"ID":"54476a5d-bb21-498c-b449-ad8f4fd2ce0d","Type":"ContainerStarted","Data":"0589e6b52cec9622f4830588b77c1692eccb19006271420bf859f5e27e3aabaf"} Jul 14 20:23:05.869707 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:05.869332 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" event={"ID":"54476a5d-bb21-498c-b449-ad8f4fd2ce0d","Type":"ContainerStarted","Data":"52a72fa2bbb61b4143f64a68a492739524046dbd1c8d4a7746743a86e80c7781"} Jul 14 20:23:05.885982 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:05.885939 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" podStartSLOduration=1.8859255529999999 podStartE2EDuration="1.885925553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:23:05.88465523 +0000 UTC m=+2203.964401022" watchObservedRunningTime="2026-07-14 20:23:05.885925553 +0000 UTC m=+2203.965671344" Jul 14 20:23:09.402256 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:09.402213 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:23:14.905715 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:14.905684 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:14.906270 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:14.905734 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:14.918530 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:14.918504 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:15.925348 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:15.925319 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:23:25.575175 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.575142 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn"] Jul 14 20:23:25.578685 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.578664 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.580791 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.580766 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2e7e45\"/\"openshift-service-ca.crt\"" Jul 14 20:23:25.580916 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.580806 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/588307a4-f275-41eb-b149-137e6f5d9766-tls-certs\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.580916 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.580837 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-home\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.580916 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.580861 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vslq\" (UniqueName: \"kubernetes.io/projected/588307a4-f275-41eb-b149-137e6f5d9766-kube-api-access-6vslq\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.581041 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.580943 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-dshm\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.581041 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.580978 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-model-cache\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.581041 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.581001 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-tmp-dir\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.581464 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.581443 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2e7e45\"/\"kube-root-ca.crt\"" Jul 14 20:23:25.581578 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.581516 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2e7e45\"/\"default-dockercfg-wqrr6\"" Jul 14 20:23:25.581578 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.581572 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2e7e45\"/\"test-model-2e7e45-kserve-self-signed-certs\"" Jul 14 20:23:25.586190 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.586167 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn"] Jul 14 20:23:25.681797 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.681761 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/588307a4-f275-41eb-b149-137e6f5d9766-tls-certs\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.681797 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.681798 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-home\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.682088 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.681825 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vslq\" (UniqueName: \"kubernetes.io/projected/588307a4-f275-41eb-b149-137e6f5d9766-kube-api-access-6vslq\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.682088 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.681862 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-dshm\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.682088 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.681888 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-model-cache\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.682088 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.681911 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-tmp-dir\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.682306 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.682289 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-model-cache\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.682357 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.682302 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-home\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.682393 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.682366 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-tmp-dir\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.684050 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.684034 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-dshm\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.684334 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.684317 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/588307a4-f275-41eb-b149-137e6f5d9766-tls-certs\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.690102 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.690076 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vslq\" (UniqueName: \"kubernetes.io/projected/588307a4-f275-41eb-b149-137e6f5d9766-kube-api-access-6vslq\") pod \"test-model-2e7e45-kserve-54bd8f69f7-b9sdn\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:25.889069 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:25.889022 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:26.020133 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:26.020035 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn"] Jul 14 20:23:26.022461 ip-10-0-132-184 kubenswrapper[2557]: W0714 20:23:26.022424 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod588307a4_f275_41eb_b149_137e6f5d9766.slice/crio-0c127935d3dcedd2fc709bc1eff8856831f1bac0a86353c3078fdc98ca3967c6 WatchSource:0}: Error finding container 0c127935d3dcedd2fc709bc1eff8856831f1bac0a86353c3078fdc98ca3967c6: Status 404 returned error can't find the container with id 0c127935d3dcedd2fc709bc1eff8856831f1bac0a86353c3078fdc98ca3967c6 Jul 14 20:23:26.963285 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:26.963244 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" event={"ID":"588307a4-f275-41eb-b149-137e6f5d9766","Type":"ContainerStarted","Data":"e52d89138c1c515c9c04b8b4e1d428d7975c9f26772fd3e03661c8aa8ed380f5"} Jul 14 20:23:26.963691 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:26.963300 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" event={"ID":"588307a4-f275-41eb-b149-137e6f5d9766","Type":"ContainerStarted","Data":"0c127935d3dcedd2fc709bc1eff8856831f1bac0a86353c3078fdc98ca3967c6"} Jul 14 20:23:26.979602 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:26.979555 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" podStartSLOduration=1.979539544 podStartE2EDuration="1.979539544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:23:26.978273963 +0000 UTC m=+2225.058019776" watchObservedRunningTime="2026-07-14 20:23:26.979539544 +0000 UTC m=+2225.059285335" Jul 14 20:23:30.791869 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:30.791830 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:23:35.889986 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:35.889949 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:35.890398 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:35.890095 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:35.902663 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:35.902637 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:36.016824 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:36.016796 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:23:48.211494 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:48.211408 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst"] Jul 14 20:23:49.104531 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:49.104487 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn"] Jul 14 20:23:56.197148 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:23:56.197110 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:24:03.231619 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.231570 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" podUID="54476a5d-bb21-498c-b449-ad8f4fd2ce0d" containerName="main" containerID="cri-o://0589e6b52cec9622f4830588b77c1692eccb19006271420bf859f5e27e3aabaf" gracePeriod=45 Jul 14 20:24:03.480488 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.480462 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:24:03.617537 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.617498 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-model-cache\") pod \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " Jul 14 20:24:03.617537 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.617549 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-dshm\") pod \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " Jul 14 20:24:03.617762 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.617580 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-home\") pod \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " Jul 14 20:24:03.617762 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.617648 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-tmp-dir\") pod \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " Jul 14 20:24:03.617762 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.617705 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-tls-certs\") pod \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " Jul 14 20:24:03.617762 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.617743 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9b49l\" (UniqueName: \"kubernetes.io/projected/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-kube-api-access-9b49l\") pod \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\" (UID: \"54476a5d-bb21-498c-b449-ad8f4fd2ce0d\") " Jul 14 20:24:03.617968 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.617818 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-model-cache" (OuterVolumeSpecName: "model-cache") pod "54476a5d-bb21-498c-b449-ad8f4fd2ce0d" (UID: "54476a5d-bb21-498c-b449-ad8f4fd2ce0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:03.617968 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.617921 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-home" (OuterVolumeSpecName: "home") pod "54476a5d-bb21-498c-b449-ad8f4fd2ce0d" (UID: "54476a5d-bb21-498c-b449-ad8f4fd2ce0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:03.617968 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.617937 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54476a5d-bb21-498c-b449-ad8f4fd2ce0d" (UID: "54476a5d-bb21-498c-b449-ad8f4fd2ce0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:03.618149 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.618102 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-home\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:24:03.618149 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.618123 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-tmp-dir\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:24:03.618149 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.618144 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-model-cache\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:24:03.619742 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.619718 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-dshm" (OuterVolumeSpecName: "dshm") pod "54476a5d-bb21-498c-b449-ad8f4fd2ce0d" (UID: "54476a5d-bb21-498c-b449-ad8f4fd2ce0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:03.619840 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.619773 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54476a5d-bb21-498c-b449-ad8f4fd2ce0d" (UID: "54476a5d-bb21-498c-b449-ad8f4fd2ce0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:24:03.619874 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.619855 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-kube-api-access-9b49l" (OuterVolumeSpecName: "kube-api-access-9b49l") pod "54476a5d-bb21-498c-b449-ad8f4fd2ce0d" (UID: "54476a5d-bb21-498c-b449-ad8f4fd2ce0d"). InnerVolumeSpecName "kube-api-access-9b49l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:24:03.718600 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.718562 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-tls-certs\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:24:03.718600 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.718593 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b49l\" (UniqueName: \"kubernetes.io/projected/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-kube-api-access-9b49l\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:24:03.718600 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:03.718605 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54476a5d-bb21-498c-b449-ad8f4fd2ce0d-dshm\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:24:04.124730 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.124668 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" podUID="588307a4-f275-41eb-b149-137e6f5d9766" containerName="main" containerID="cri-o://e52d89138c1c515c9c04b8b4e1d428d7975c9f26772fd3e03661c8aa8ed380f5" gracePeriod=45 Jul 14 20:24:04.136758 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.136730 2557 generic.go:356] "Generic (PLEG): container finished" podID="54476a5d-bb21-498c-b449-ad8f4fd2ce0d" containerID="0589e6b52cec9622f4830588b77c1692eccb19006271420bf859f5e27e3aabaf" exitCode=0 Jul 14 20:24:04.136872 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.136766 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" Jul 14 20:24:04.136872 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.136809 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" event={"ID":"54476a5d-bb21-498c-b449-ad8f4fd2ce0d","Type":"ContainerDied","Data":"0589e6b52cec9622f4830588b77c1692eccb19006271420bf859f5e27e3aabaf"} Jul 14 20:24:04.136872 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.136854 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst" event={"ID":"54476a5d-bb21-498c-b449-ad8f4fd2ce0d","Type":"ContainerDied","Data":"52a72fa2bbb61b4143f64a68a492739524046dbd1c8d4a7746743a86e80c7781"} Jul 14 20:24:04.136872 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.136872 2557 scope.go:117] "RemoveContainer" containerID="0589e6b52cec9622f4830588b77c1692eccb19006271420bf859f5e27e3aabaf" Jul 14 20:24:04.196914 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.196883 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst"] Jul 14 20:24:04.201385 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.201359 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1c67fa/test-model-1c67fa-kserve-58d47fd677-zvpst"] Jul 14 20:24:04.234846 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.234705 2557 scope.go:117] "RemoveContainer" containerID="0589e6b52cec9622f4830588b77c1692eccb19006271420bf859f5e27e3aabaf" Jul 14 20:24:04.235128 ip-10-0-132-184 kubenswrapper[2557]: E0714 20:24:04.235034 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0589e6b52cec9622f4830588b77c1692eccb19006271420bf859f5e27e3aabaf\": container with ID starting with 0589e6b52cec9622f4830588b77c1692eccb19006271420bf859f5e27e3aabaf not found: ID does not exist" containerID="0589e6b52cec9622f4830588b77c1692eccb19006271420bf859f5e27e3aabaf" Jul 14 20:24:04.235128 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.235083 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0589e6b52cec9622f4830588b77c1692eccb19006271420bf859f5e27e3aabaf"} err="failed to get container status \"0589e6b52cec9622f4830588b77c1692eccb19006271420bf859f5e27e3aabaf\": rpc error: code = NotFound desc = could not find container \"0589e6b52cec9622f4830588b77c1692eccb19006271420bf859f5e27e3aabaf\": container with ID starting with 0589e6b52cec9622f4830588b77c1692eccb19006271420bf859f5e27e3aabaf not found: ID does not exist" Jul 14 20:24:04.362542 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.362503 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54476a5d-bb21-498c-b449-ad8f4fd2ce0d" path="/var/lib/kubelet/pods/54476a5d-bb21-498c-b449-ad8f4fd2ce0d/volumes" Jul 14 20:24:04.381911 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.381859 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:24:04.527737 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.527702 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/588307a4-f275-41eb-b149-137e6f5d9766-tls-certs\") pod \"588307a4-f275-41eb-b149-137e6f5d9766\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " Jul 14 20:24:04.527902 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.527798 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vslq\" (UniqueName: \"kubernetes.io/projected/588307a4-f275-41eb-b149-137e6f5d9766-kube-api-access-6vslq\") pod \"588307a4-f275-41eb-b149-137e6f5d9766\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " Jul 14 20:24:04.527902 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.527820 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-model-cache\") pod \"588307a4-f275-41eb-b149-137e6f5d9766\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " Jul 14 20:24:04.527902 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.527850 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-tmp-dir\") pod \"588307a4-f275-41eb-b149-137e6f5d9766\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " Jul 14 20:24:04.527902 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.527876 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-home\") pod \"588307a4-f275-41eb-b149-137e6f5d9766\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " Jul 14 20:24:04.528129 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.527918 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-dshm\") pod \"588307a4-f275-41eb-b149-137e6f5d9766\" (UID: \"588307a4-f275-41eb-b149-137e6f5d9766\") " Jul 14 20:24:04.528178 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.528130 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-model-cache" (OuterVolumeSpecName: "model-cache") pod "588307a4-f275-41eb-b149-137e6f5d9766" (UID: "588307a4-f275-41eb-b149-137e6f5d9766"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:04.528178 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.528165 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "588307a4-f275-41eb-b149-137e6f5d9766" (UID: "588307a4-f275-41eb-b149-137e6f5d9766"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:04.528312 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.528273 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-home" (OuterVolumeSpecName: "home") pod "588307a4-f275-41eb-b149-137e6f5d9766" (UID: "588307a4-f275-41eb-b149-137e6f5d9766"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:04.528384 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.528341 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-model-cache\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:24:04.528384 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.528378 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-tmp-dir\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:24:04.528493 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.528397 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-home\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:24:04.530214 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.530180 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-dshm" (OuterVolumeSpecName: "dshm") pod "588307a4-f275-41eb-b149-137e6f5d9766" (UID: "588307a4-f275-41eb-b149-137e6f5d9766"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:24:04.530304 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.530257 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/588307a4-f275-41eb-b149-137e6f5d9766-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "588307a4-f275-41eb-b149-137e6f5d9766" (UID: "588307a4-f275-41eb-b149-137e6f5d9766"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:24:04.530304 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.530265 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588307a4-f275-41eb-b149-137e6f5d9766-kube-api-access-6vslq" (OuterVolumeSpecName: "kube-api-access-6vslq") pod "588307a4-f275-41eb-b149-137e6f5d9766" (UID: "588307a4-f275-41eb-b149-137e6f5d9766"). InnerVolumeSpecName "kube-api-access-6vslq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:24:04.629042 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.628987 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vslq\" (UniqueName: \"kubernetes.io/projected/588307a4-f275-41eb-b149-137e6f5d9766-kube-api-access-6vslq\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:24:04.629042 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.629034 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/588307a4-f275-41eb-b149-137e6f5d9766-dshm\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:24:04.629042 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:04.629049 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/588307a4-f275-41eb-b149-137e6f5d9766-tls-certs\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:24:05.143533 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:05.143497 2557 generic.go:356] "Generic (PLEG): container finished" podID="588307a4-f275-41eb-b149-137e6f5d9766" containerID="e52d89138c1c515c9c04b8b4e1d428d7975c9f26772fd3e03661c8aa8ed380f5" exitCode=0 Jul 14 20:24:05.143703 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:05.143546 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" Jul 14 20:24:05.143703 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:05.143544 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" event={"ID":"588307a4-f275-41eb-b149-137e6f5d9766","Type":"ContainerDied","Data":"e52d89138c1c515c9c04b8b4e1d428d7975c9f26772fd3e03661c8aa8ed380f5"} Jul 14 20:24:05.143703 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:05.143656 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn" event={"ID":"588307a4-f275-41eb-b149-137e6f5d9766","Type":"ContainerDied","Data":"0c127935d3dcedd2fc709bc1eff8856831f1bac0a86353c3078fdc98ca3967c6"} Jul 14 20:24:05.143703 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:05.143681 2557 scope.go:117] "RemoveContainer" containerID="e52d89138c1c515c9c04b8b4e1d428d7975c9f26772fd3e03661c8aa8ed380f5" Jul 14 20:24:05.152366 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:05.152341 2557 scope.go:117] "RemoveContainer" containerID="e52d89138c1c515c9c04b8b4e1d428d7975c9f26772fd3e03661c8aa8ed380f5" Jul 14 20:24:05.152624 ip-10-0-132-184 kubenswrapper[2557]: E0714 20:24:05.152601 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52d89138c1c515c9c04b8b4e1d428d7975c9f26772fd3e03661c8aa8ed380f5\": container with ID starting with e52d89138c1c515c9c04b8b4e1d428d7975c9f26772fd3e03661c8aa8ed380f5 not found: ID does not exist" containerID="e52d89138c1c515c9c04b8b4e1d428d7975c9f26772fd3e03661c8aa8ed380f5" Jul 14 20:24:05.152670 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:05.152633 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52d89138c1c515c9c04b8b4e1d428d7975c9f26772fd3e03661c8aa8ed380f5"} err="failed to get container status \"e52d89138c1c515c9c04b8b4e1d428d7975c9f26772fd3e03661c8aa8ed380f5\": rpc error: code = NotFound desc = could not find container \"e52d89138c1c515c9c04b8b4e1d428d7975c9f26772fd3e03661c8aa8ed380f5\": container with ID starting with e52d89138c1c515c9c04b8b4e1d428d7975c9f26772fd3e03661c8aa8ed380f5 not found: ID does not exist" Jul 14 20:24:05.164388 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:05.164363 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn"] Jul 14 20:24:05.168394 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:05.168373 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2e7e45/test-model-2e7e45-kserve-54bd8f69f7-b9sdn"] Jul 14 20:24:06.360988 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:06.360944 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="588307a4-f275-41eb-b149-137e6f5d9766" path="/var/lib/kubelet/pods/588307a4-f275-41eb-b149-137e6f5d9766/volumes" Jul 14 20:24:10.988481 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:10.988442 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:24:28.984488 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:28.984457 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:24:49.798732 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:24:49.798691 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:25:00.404791 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:25:00.404757 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:25:19.296983 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:25:19.296897 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:25:45.095499 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:25:45.095461 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:26:06.487974 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:06.487935 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:26:19.189683 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:19.189637 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:26:22.456479 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:22.456451 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 20:26:22.461817 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:22.461795 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 20:26:27.204483 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.204452 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw"] Jul 14 20:26:27.204924 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.204784 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="588307a4-f275-41eb-b149-137e6f5d9766" containerName="main" Jul 14 20:26:27.204924 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.204795 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="588307a4-f275-41eb-b149-137e6f5d9766" containerName="main" Jul 14 20:26:27.204924 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.204811 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54476a5d-bb21-498c-b449-ad8f4fd2ce0d" containerName="main" Jul 14 20:26:27.204924 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.204817 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="54476a5d-bb21-498c-b449-ad8f4fd2ce0d" containerName="main" Jul 14 20:26:27.204924 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.204868 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="588307a4-f275-41eb-b149-137e6f5d9766" containerName="main" Jul 14 20:26:27.204924 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.204875 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="54476a5d-bb21-498c-b449-ad8f4fd2ce0d" containerName="main" Jul 14 20:26:27.207850 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.207831 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.210908 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.210883 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fad346\"/\"kube-root-ca.crt\"" Jul 14 20:26:27.210908 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.210899 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fad346\"/\"auth-test-model-fad346-kserve-self-signed-certs\"" Jul 14 20:26:27.211133 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.210883 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fad346\"/\"default-dockercfg-pcpdr\"" Jul 14 20:26:27.211133 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.210970 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fad346\"/\"openshift-service-ca.crt\"" Jul 14 20:26:27.218852 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.218827 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw"] Jul 14 20:26:27.382592 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.382557 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac438432-2c13-405e-92b2-c30e458041a4-tls-certs\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.382811 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.382608 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-model-cache\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.382811 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.382631 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mtgt\" (UniqueName: \"kubernetes.io/projected/ac438432-2c13-405e-92b2-c30e458041a4-kube-api-access-7mtgt\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.382811 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.382710 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-tmp-dir\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.382811 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.382757 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-dshm\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.382986 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.382821 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-home\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.483455 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.483361 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-dshm\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.483455 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.483410 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-home\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.483455 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.483451 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac438432-2c13-405e-92b2-c30e458041a4-tls-certs\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.483727 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.483480 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-model-cache\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.483727 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.483504 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mtgt\" (UniqueName: \"kubernetes.io/projected/ac438432-2c13-405e-92b2-c30e458041a4-kube-api-access-7mtgt\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.483727 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.483526 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-tmp-dir\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.483961 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.483907 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-model-cache\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.483961 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.483927 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-home\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.484077 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.483982 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-tmp-dir\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.485669 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.485649 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-dshm\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.485953 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.485938 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac438432-2c13-405e-92b2-c30e458041a4-tls-certs\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.492275 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.492254 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mtgt\" (UniqueName: \"kubernetes.io/projected/ac438432-2c13-405e-92b2-c30e458041a4-kube-api-access-7mtgt\") pod \"auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.517880 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.517849 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:27.646748 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.646721 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw"] Jul 14 20:26:27.649518 ip-10-0-132-184 kubenswrapper[2557]: W0714 20:26:27.649487 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac438432_2c13_405e_92b2_c30e458041a4.slice/crio-9ebaf22f673e33e0e1ff9c62c0413ce83fe98c974dd4fb4dfcfb04a2e3fe8eb1 WatchSource:0}: Error finding container 9ebaf22f673e33e0e1ff9c62c0413ce83fe98c974dd4fb4dfcfb04a2e3fe8eb1: Status 404 returned error can't find the container with id 9ebaf22f673e33e0e1ff9c62c0413ce83fe98c974dd4fb4dfcfb04a2e3fe8eb1 Jul 14 20:26:27.767943 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.767828 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" event={"ID":"ac438432-2c13-405e-92b2-c30e458041a4","Type":"ContainerStarted","Data":"db0a6d0bc9da8baf8cf5d6723928615c11c2e0c11dfbcd3e478c793ffccb6412"} Jul 14 20:26:27.767943 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.767877 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" event={"ID":"ac438432-2c13-405e-92b2-c30e458041a4","Type":"ContainerStarted","Data":"9ebaf22f673e33e0e1ff9c62c0413ce83fe98c974dd4fb4dfcfb04a2e3fe8eb1"} Jul 14 20:26:27.785045 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:27.784995 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" podStartSLOduration=0.784978172 podStartE2EDuration="784.978172ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:26:27.783959952 +0000 UTC m=+2405.863705769" watchObservedRunningTime="2026-07-14 20:26:27.784978172 +0000 UTC m=+2405.864723963" Jul 14 20:26:31.903897 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:31.903860 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:26:37.518157 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:37.518122 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:37.518157 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:37.518178 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:37.530493 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:37.530460 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:37.821176 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:37.821090 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:26:56.495469 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:26:56.495386 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:27:18.191210 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:27:18.191172 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:27:36.194614 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:27:36.194583 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:27:50.593421 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:27:50.593386 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:28:05.994088 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:28:05.994034 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:28:23.305793 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:28:23.305703 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:28:42.303262 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:28:42.303225 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:29:00.393654 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:00.393620 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:29:05.273373 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:05.273341 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw"] Jul 14 20:29:15.803955 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.803916 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp"] Jul 14 20:29:15.809555 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.809509 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.812494 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.812469 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fad346\"/\"sub-test-model-fad346-kserve-self-signed-certs\"" Jul 14 20:29:15.815733 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.815705 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp"] Jul 14 20:29:15.891995 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.891955 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-home\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.892209 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.892020 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nptz\" (UniqueName: \"kubernetes.io/projected/fa9457c0-720a-4920-a10f-d479af6978de-kube-api-access-5nptz\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.892209 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.892094 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-model-cache\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.892209 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.892131 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa9457c0-720a-4920-a10f-d479af6978de-tls-certs\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.892209 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.892169 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-tmp-dir\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.892372 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.892216 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-dshm\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.992783 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.992743 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nptz\" (UniqueName: \"kubernetes.io/projected/fa9457c0-720a-4920-a10f-d479af6978de-kube-api-access-5nptz\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.992962 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.992796 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-model-cache\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.992962 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.992815 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa9457c0-720a-4920-a10f-d479af6978de-tls-certs\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.992962 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.992830 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-tmp-dir\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.992962 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.992870 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-dshm\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.992962 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.992921 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-home\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.993270 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.993225 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-model-cache\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.993270 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.993242 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-tmp-dir\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.993351 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.993299 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-home\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.995082 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.995047 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-dshm\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:15.995487 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:15.995468 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa9457c0-720a-4920-a10f-d479af6978de-tls-certs\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:16.002383 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:16.002357 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nptz\" (UniqueName: \"kubernetes.io/projected/fa9457c0-720a-4920-a10f-d479af6978de-kube-api-access-5nptz\") pod \"sub-test-model-fad346-kserve-66fcdd964c-cf4dp\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:16.121119 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:16.121089 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:16.254525 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:16.254499 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp"] Jul 14 20:29:16.256824 ip-10-0-132-184 kubenswrapper[2557]: W0714 20:29:16.256798 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa9457c0_720a_4920_a10f_d479af6978de.slice/crio-a52aa28701a5304df44aea99e471b62f1f6286eecbac9dfb001385947079994f WatchSource:0}: Error finding container a52aa28701a5304df44aea99e471b62f1f6286eecbac9dfb001385947079994f: Status 404 returned error can't find the container with id a52aa28701a5304df44aea99e471b62f1f6286eecbac9dfb001385947079994f Jul 14 20:29:16.515171 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:16.515050 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" event={"ID":"fa9457c0-720a-4920-a10f-d479af6978de","Type":"ContainerStarted","Data":"dc87e3e7aead753f665b27e9234c9aa3a87157a707d92313d0ae62b205d3c107"} Jul 14 20:29:16.515171 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:16.515129 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" event={"ID":"fa9457c0-720a-4920-a10f-d479af6978de","Type":"ContainerStarted","Data":"a52aa28701a5304df44aea99e471b62f1f6286eecbac9dfb001385947079994f"} Jul 14 20:29:16.535489 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:16.535439 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" podStartSLOduration=1.535424941 podStartE2EDuration="1.535424941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:29:16.533204574 +0000 UTC m=+2574.612950367" watchObservedRunningTime="2026-07-14 20:29:16.535424941 +0000 UTC m=+2574.615170769" Jul 14 20:29:19.996361 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:19.996310 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:29:20.293191 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.293108 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" podUID="ac438432-2c13-405e-92b2-c30e458041a4" containerName="main" containerID="cri-o://db0a6d0bc9da8baf8cf5d6723928615c11c2e0c11dfbcd3e478c793ffccb6412" gracePeriod=45 Jul 14 20:29:20.534743 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.534714 2557 generic.go:356] "Generic (PLEG): container finished" podID="ac438432-2c13-405e-92b2-c30e458041a4" containerID="db0a6d0bc9da8baf8cf5d6723928615c11c2e0c11dfbcd3e478c793ffccb6412" exitCode=0 Jul 14 20:29:20.534822 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.534786 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" event={"ID":"ac438432-2c13-405e-92b2-c30e458041a4","Type":"ContainerDied","Data":"db0a6d0bc9da8baf8cf5d6723928615c11c2e0c11dfbcd3e478c793ffccb6412"} Jul 14 20:29:20.534861 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.534839 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" event={"ID":"ac438432-2c13-405e-92b2-c30e458041a4","Type":"ContainerDied","Data":"9ebaf22f673e33e0e1ff9c62c0413ce83fe98c974dd4fb4dfcfb04a2e3fe8eb1"} Jul 14 20:29:20.534895 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.534857 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ebaf22f673e33e0e1ff9c62c0413ce83fe98c974dd4fb4dfcfb04a2e3fe8eb1" Jul 14 20:29:20.550565 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.550498 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:29:20.636780 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.636737 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mtgt\" (UniqueName: \"kubernetes.io/projected/ac438432-2c13-405e-92b2-c30e458041a4-kube-api-access-7mtgt\") pod \"ac438432-2c13-405e-92b2-c30e458041a4\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " Jul 14 20:29:20.636780 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.636782 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac438432-2c13-405e-92b2-c30e458041a4-tls-certs\") pod \"ac438432-2c13-405e-92b2-c30e458041a4\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " Jul 14 20:29:20.637037 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.636806 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-tmp-dir\") pod \"ac438432-2c13-405e-92b2-c30e458041a4\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " Jul 14 20:29:20.637037 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.636848 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-home\") pod \"ac438432-2c13-405e-92b2-c30e458041a4\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " Jul 14 20:29:20.637037 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.636875 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-model-cache\") pod \"ac438432-2c13-405e-92b2-c30e458041a4\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " Jul 14 20:29:20.637037 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.636900 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-dshm\") pod \"ac438432-2c13-405e-92b2-c30e458041a4\" (UID: \"ac438432-2c13-405e-92b2-c30e458041a4\") " Jul 14 20:29:20.637252 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.637096 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac438432-2c13-405e-92b2-c30e458041a4" (UID: "ac438432-2c13-405e-92b2-c30e458041a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:29:20.637306 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.637257 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-tmp-dir\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:29:20.637460 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.637405 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac438432-2c13-405e-92b2-c30e458041a4" (UID: "ac438432-2c13-405e-92b2-c30e458041a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:29:20.637585 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.637564 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-home" (OuterVolumeSpecName: "home") pod "ac438432-2c13-405e-92b2-c30e458041a4" (UID: "ac438432-2c13-405e-92b2-c30e458041a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:29:20.639417 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.639391 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-dshm" (OuterVolumeSpecName: "dshm") pod "ac438432-2c13-405e-92b2-c30e458041a4" (UID: "ac438432-2c13-405e-92b2-c30e458041a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:29:20.639538 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.639516 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac438432-2c13-405e-92b2-c30e458041a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac438432-2c13-405e-92b2-c30e458041a4" (UID: "ac438432-2c13-405e-92b2-c30e458041a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:29:20.639613 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.639535 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac438432-2c13-405e-92b2-c30e458041a4-kube-api-access-7mtgt" (OuterVolumeSpecName: "kube-api-access-7mtgt") pod "ac438432-2c13-405e-92b2-c30e458041a4" (UID: "ac438432-2c13-405e-92b2-c30e458041a4"). InnerVolumeSpecName "kube-api-access-7mtgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:29:20.738107 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.738039 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mtgt\" (UniqueName: \"kubernetes.io/projected/ac438432-2c13-405e-92b2-c30e458041a4-kube-api-access-7mtgt\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:29:20.738107 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.738102 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac438432-2c13-405e-92b2-c30e458041a4-tls-certs\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:29:20.738107 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.738115 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-home\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:29:20.738348 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.738126 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-model-cache\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:29:20.738348 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:20.738138 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac438432-2c13-405e-92b2-c30e458041a4-dshm\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:29:21.539490 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:21.539455 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw" Jul 14 20:29:21.565215 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:21.564886 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw"] Jul 14 20:29:21.565215 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:21.564937 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fad346/auth-test-model-fad346-kserve-57dbdc5dc5-s4wmw"] Jul 14 20:29:22.360695 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:22.360662 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac438432-2c13-405e-92b2-c30e458041a4" path="/var/lib/kubelet/pods/ac438432-2c13-405e-92b2-c30e458041a4/volumes" Jul 14 20:29:26.121653 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:26.121622 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:26.122044 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:26.121668 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:26.133722 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:26.133700 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:26.571253 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:26.571166 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:29:37.795615 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.795581 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc"] Jul 14 20:29:37.796173 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.795987 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac438432-2c13-405e-92b2-c30e458041a4" containerName="main" Jul 14 20:29:37.796173 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.796002 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac438432-2c13-405e-92b2-c30e458041a4" containerName="main" Jul 14 20:29:37.796173 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.796088 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac438432-2c13-405e-92b2-c30e458041a4" containerName="main" Jul 14 20:29:37.798161 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.798141 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.800405 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.800385 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a99cbd\"/\"kube-root-ca.crt\"" Jul 14 20:29:37.801082 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.801043 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a99cbd\"/\"default-dockercfg-5fhmn\"" Jul 14 20:29:37.801301 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.801284 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a99cbd\"/\"openshift-service-ca.crt\"" Jul 14 20:29:37.801371 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.801309 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a99cbd\"/\"sub-test-model-a99cbd-kserve-self-signed-certs\"" Jul 14 20:29:37.807699 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.807678 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc"] Jul 14 20:29:37.890218 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.890187 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-tmp-dir\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.890400 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.890231 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bwml\" (UniqueName: \"kubernetes.io/projected/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-kube-api-access-9bwml\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.890400 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.890254 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-model-cache\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.890400 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.890275 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-tls-certs\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.890400 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.890321 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-home\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.890533 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.890429 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-dshm\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.991149 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.991114 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-tmp-dir\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.991149 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.991157 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bwml\" (UniqueName: \"kubernetes.io/projected/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-kube-api-access-9bwml\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.991411 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.991176 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-model-cache\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.991411 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.991194 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-tls-certs\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.991411 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.991212 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-home\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.991411 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.991313 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-dshm\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.991610 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.991585 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-tmp-dir\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.991610 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.991604 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-home\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.991896 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.991875 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-model-cache\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.993604 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.993582 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-dshm\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.993686 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.993670 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-tls-certs\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:37.999772 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:37.999753 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bwml\" (UniqueName: \"kubernetes.io/projected/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-kube-api-access-9bwml\") pod \"sub-test-model-a99cbd-kserve-867b86cc5-qxzwc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:38.107627 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:38.107594 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:38.239870 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:38.239840 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc"] Jul 14 20:29:38.242266 ip-10-0-132-184 kubenswrapper[2557]: W0714 20:29:38.242236 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f3338cc_96fb_4a33_9934_fbdc3d22cafc.slice/crio-3df3eab59e6cbe5ab09009d858e857955f61326d7726b34f1796c1fe18f74710 WatchSource:0}: Error finding container 3df3eab59e6cbe5ab09009d858e857955f61326d7726b34f1796c1fe18f74710: Status 404 returned error can't find the container with id 3df3eab59e6cbe5ab09009d858e857955f61326d7726b34f1796c1fe18f74710 Jul 14 20:29:38.615482 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:38.615439 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" event={"ID":"6f3338cc-96fb-4a33-9934-fbdc3d22cafc","Type":"ContainerStarted","Data":"45a8c621bb9cd6ae0af4250532e307bd603e88628a570b5300cb0eca73aba035"} Jul 14 20:29:38.615650 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:38.615492 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" event={"ID":"6f3338cc-96fb-4a33-9934-fbdc3d22cafc","Type":"ContainerStarted","Data":"3df3eab59e6cbe5ab09009d858e857955f61326d7726b34f1796c1fe18f74710"} Jul 14 20:29:38.632487 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:38.632431 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" podStartSLOduration=1.632413659 podStartE2EDuration="1.632413659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:29:38.632106209 +0000 UTC m=+2596.711852003" watchObservedRunningTime="2026-07-14 20:29:38.632413659 +0000 UTC m=+2596.712159451" Jul 14 20:29:41.999590 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:41.999551 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:29:48.108583 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:48.108491 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:48.108583 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:48.108550 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:48.120960 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:48.120931 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:29:48.674032 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:29:48.673998 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:30:01.704726 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:01.704680 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:30:21.396483 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:21.396445 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:30:27.546839 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:27.546802 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp"] Jul 14 20:30:28.481111 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:28.481073 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc"] Jul 14 20:30:41.510353 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:41.510313 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:30:42.566955 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.566912 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" podUID="fa9457c0-720a-4920-a10f-d479af6978de" containerName="main" containerID="cri-o://dc87e3e7aead753f665b27e9234c9aa3a87157a707d92313d0ae62b205d3c107" gracePeriod=45 Jul 14 20:30:42.814900 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.814875 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:30:42.892231 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.892195 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-home\") pod \"fa9457c0-720a-4920-a10f-d479af6978de\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " Jul 14 20:30:42.892419 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.892262 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-dshm\") pod \"fa9457c0-720a-4920-a10f-d479af6978de\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " Jul 14 20:30:42.892419 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.892288 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nptz\" (UniqueName: \"kubernetes.io/projected/fa9457c0-720a-4920-a10f-d479af6978de-kube-api-access-5nptz\") pod \"fa9457c0-720a-4920-a10f-d479af6978de\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " Jul 14 20:30:42.892419 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.892305 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa9457c0-720a-4920-a10f-d479af6978de-tls-certs\") pod \"fa9457c0-720a-4920-a10f-d479af6978de\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " Jul 14 20:30:42.892419 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.892329 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-model-cache\") pod \"fa9457c0-720a-4920-a10f-d479af6978de\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " Jul 14 20:30:42.892419 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.892389 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-tmp-dir\") pod \"fa9457c0-720a-4920-a10f-d479af6978de\" (UID: \"fa9457c0-720a-4920-a10f-d479af6978de\") " Jul 14 20:30:42.892681 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.892473 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-home" (OuterVolumeSpecName: "home") pod "fa9457c0-720a-4920-a10f-d479af6978de" (UID: "fa9457c0-720a-4920-a10f-d479af6978de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:30:42.892681 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.892658 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa9457c0-720a-4920-a10f-d479af6978de" (UID: "fa9457c0-720a-4920-a10f-d479af6978de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:30:42.892781 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.892675 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa9457c0-720a-4920-a10f-d479af6978de" (UID: "fa9457c0-720a-4920-a10f-d479af6978de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:30:42.892831 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.892777 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-home\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:30:42.894510 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.894481 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9457c0-720a-4920-a10f-d479af6978de-kube-api-access-5nptz" (OuterVolumeSpecName: "kube-api-access-5nptz") pod "fa9457c0-720a-4920-a10f-d479af6978de" (UID: "fa9457c0-720a-4920-a10f-d479af6978de"). InnerVolumeSpecName "kube-api-access-5nptz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:30:42.894858 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.894836 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-dshm" (OuterVolumeSpecName: "dshm") pod "fa9457c0-720a-4920-a10f-d479af6978de" (UID: "fa9457c0-720a-4920-a10f-d479af6978de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:30:42.894921 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.894841 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa9457c0-720a-4920-a10f-d479af6978de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa9457c0-720a-4920-a10f-d479af6978de" (UID: "fa9457c0-720a-4920-a10f-d479af6978de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:30:42.910330 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.910302 2557 generic.go:356] "Generic (PLEG): container finished" podID="fa9457c0-720a-4920-a10f-d479af6978de" containerID="dc87e3e7aead753f665b27e9234c9aa3a87157a707d92313d0ae62b205d3c107" exitCode=0 Jul 14 20:30:42.910479 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.910346 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" Jul 14 20:30:42.910479 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.910347 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" event={"ID":"fa9457c0-720a-4920-a10f-d479af6978de","Type":"ContainerDied","Data":"dc87e3e7aead753f665b27e9234c9aa3a87157a707d92313d0ae62b205d3c107"} Jul 14 20:30:42.910588 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.910477 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp" event={"ID":"fa9457c0-720a-4920-a10f-d479af6978de","Type":"ContainerDied","Data":"a52aa28701a5304df44aea99e471b62f1f6286eecbac9dfb001385947079994f"} Jul 14 20:30:42.910588 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.910509 2557 scope.go:117] "RemoveContainer" containerID="dc87e3e7aead753f665b27e9234c9aa3a87157a707d92313d0ae62b205d3c107" Jul 14 20:30:42.919385 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.919366 2557 scope.go:117] "RemoveContainer" containerID="dc87e3e7aead753f665b27e9234c9aa3a87157a707d92313d0ae62b205d3c107" Jul 14 20:30:42.919670 ip-10-0-132-184 kubenswrapper[2557]: E0714 20:30:42.919650 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc87e3e7aead753f665b27e9234c9aa3a87157a707d92313d0ae62b205d3c107\": container with ID starting with dc87e3e7aead753f665b27e9234c9aa3a87157a707d92313d0ae62b205d3c107 not found: ID does not exist" containerID="dc87e3e7aead753f665b27e9234c9aa3a87157a707d92313d0ae62b205d3c107" Jul 14 20:30:42.919737 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.919679 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc87e3e7aead753f665b27e9234c9aa3a87157a707d92313d0ae62b205d3c107"} err="failed to get container status \"dc87e3e7aead753f665b27e9234c9aa3a87157a707d92313d0ae62b205d3c107\": rpc error: code = NotFound desc = could not find container \"dc87e3e7aead753f665b27e9234c9aa3a87157a707d92313d0ae62b205d3c107\": container with ID starting with dc87e3e7aead753f665b27e9234c9aa3a87157a707d92313d0ae62b205d3c107 not found: ID does not exist" Jul 14 20:30:42.943135 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.943108 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp"] Jul 14 20:30:42.945364 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.945338 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fad346/sub-test-model-fad346-kserve-66fcdd964c-cf4dp"] Jul 14 20:30:42.993319 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.993288 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-dshm\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:30:42.993319 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.993314 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nptz\" (UniqueName: \"kubernetes.io/projected/fa9457c0-720a-4920-a10f-d479af6978de-kube-api-access-5nptz\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:30:42.993319 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.993325 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa9457c0-720a-4920-a10f-d479af6978de-tls-certs\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:30:42.993584 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.993333 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-model-cache\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:30:42.993584 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:42.993340 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa9457c0-720a-4920-a10f-d479af6978de-tmp-dir\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:30:43.501217 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.501175 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" podUID="6f3338cc-96fb-4a33-9934-fbdc3d22cafc" containerName="main" containerID="cri-o://45a8c621bb9cd6ae0af4250532e307bd603e88628a570b5300cb0eca73aba035" gracePeriod=45 Jul 14 20:30:43.752366 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.752300 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:30:43.901918 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.901887 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bwml\" (UniqueName: \"kubernetes.io/projected/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-kube-api-access-9bwml\") pod \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " Jul 14 20:30:43.901918 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.901927 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-model-cache\") pod \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " Jul 14 20:30:43.902232 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.901983 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-dshm\") pod \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " Jul 14 20:30:43.902232 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.902002 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-home\") pod \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " Jul 14 20:30:43.902232 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.902027 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-tmp-dir\") pod \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " Jul 14 20:30:43.902232 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.902100 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-tls-certs\") pod \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\" (UID: \"6f3338cc-96fb-4a33-9934-fbdc3d22cafc\") " Jul 14 20:30:43.902430 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.902345 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-home" (OuterVolumeSpecName: "home") pod "6f3338cc-96fb-4a33-9934-fbdc3d22cafc" (UID: "6f3338cc-96fb-4a33-9934-fbdc3d22cafc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:30:43.902430 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.902361 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f3338cc-96fb-4a33-9934-fbdc3d22cafc" (UID: "6f3338cc-96fb-4a33-9934-fbdc3d22cafc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:30:43.902430 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.902371 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f3338cc-96fb-4a33-9934-fbdc3d22cafc" (UID: "6f3338cc-96fb-4a33-9934-fbdc3d22cafc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:30:43.904328 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.904299 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-kube-api-access-9bwml" (OuterVolumeSpecName: "kube-api-access-9bwml") pod "6f3338cc-96fb-4a33-9934-fbdc3d22cafc" (UID: "6f3338cc-96fb-4a33-9934-fbdc3d22cafc"). InnerVolumeSpecName "kube-api-access-9bwml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:30:43.904441 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.904376 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f3338cc-96fb-4a33-9934-fbdc3d22cafc" (UID: "6f3338cc-96fb-4a33-9934-fbdc3d22cafc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:30:43.904441 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.904386 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-dshm" (OuterVolumeSpecName: "dshm") pod "6f3338cc-96fb-4a33-9934-fbdc3d22cafc" (UID: "6f3338cc-96fb-4a33-9934-fbdc3d22cafc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:30:43.917628 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.917600 2557 generic.go:356] "Generic (PLEG): container finished" podID="6f3338cc-96fb-4a33-9934-fbdc3d22cafc" containerID="45a8c621bb9cd6ae0af4250532e307bd603e88628a570b5300cb0eca73aba035" exitCode=0 Jul 14 20:30:43.917797 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.917642 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" Jul 14 20:30:43.917797 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.917674 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" event={"ID":"6f3338cc-96fb-4a33-9934-fbdc3d22cafc","Type":"ContainerDied","Data":"45a8c621bb9cd6ae0af4250532e307bd603e88628a570b5300cb0eca73aba035"} Jul 14 20:30:43.917797 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.917716 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc" event={"ID":"6f3338cc-96fb-4a33-9934-fbdc3d22cafc","Type":"ContainerDied","Data":"3df3eab59e6cbe5ab09009d858e857955f61326d7726b34f1796c1fe18f74710"} Jul 14 20:30:43.917797 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.917734 2557 scope.go:117] "RemoveContainer" containerID="45a8c621bb9cd6ae0af4250532e307bd603e88628a570b5300cb0eca73aba035" Jul 14 20:30:43.926277 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.926257 2557 scope.go:117] "RemoveContainer" containerID="45a8c621bb9cd6ae0af4250532e307bd603e88628a570b5300cb0eca73aba035" Jul 14 20:30:43.926538 ip-10-0-132-184 kubenswrapper[2557]: E0714 20:30:43.926521 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a8c621bb9cd6ae0af4250532e307bd603e88628a570b5300cb0eca73aba035\": container with ID starting with 45a8c621bb9cd6ae0af4250532e307bd603e88628a570b5300cb0eca73aba035 not found: ID does not exist" containerID="45a8c621bb9cd6ae0af4250532e307bd603e88628a570b5300cb0eca73aba035" Jul 14 20:30:43.926583 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.926547 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a8c621bb9cd6ae0af4250532e307bd603e88628a570b5300cb0eca73aba035"} err="failed to get container status \"45a8c621bb9cd6ae0af4250532e307bd603e88628a570b5300cb0eca73aba035\": rpc error: code = NotFound desc = could not find container \"45a8c621bb9cd6ae0af4250532e307bd603e88628a570b5300cb0eca73aba035\": container with ID starting with 45a8c621bb9cd6ae0af4250532e307bd603e88628a570b5300cb0eca73aba035 not found: ID does not exist" Jul 14 20:30:43.939136 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.939109 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc"] Jul 14 20:30:43.943800 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:43.943780 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/sub-test-model-a99cbd-kserve-867b86cc5-qxzwc"] Jul 14 20:30:44.002900 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:44.002814 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bwml\" (UniqueName: \"kubernetes.io/projected/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-kube-api-access-9bwml\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:30:44.002900 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:44.002842 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-model-cache\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:30:44.002900 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:44.002852 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-dshm\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:30:44.002900 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:44.002860 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-home\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:30:44.002900 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:44.002867 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-tmp-dir\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:30:44.002900 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:44.002875 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f3338cc-96fb-4a33-9934-fbdc3d22cafc-tls-certs\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:30:44.364350 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:44.362624 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f3338cc-96fb-4a33-9934-fbdc3d22cafc" path="/var/lib/kubelet/pods/6f3338cc-96fb-4a33-9934-fbdc3d22cafc/volumes" Jul 14 20:30:44.364350 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:30:44.363284 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa9457c0-720a-4920-a10f-d479af6978de" path="/var/lib/kubelet/pods/fa9457c0-720a-4920-a10f-d479af6978de/volumes" Jul 14 20:31:11.401666 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.401634 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v"] Jul 14 20:31:11.402201 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.401960 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa9457c0-720a-4920-a10f-d479af6978de" containerName="main" Jul 14 20:31:11.402201 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.401971 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa9457c0-720a-4920-a10f-d479af6978de" containerName="main" Jul 14 20:31:11.402201 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.401984 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f3338cc-96fb-4a33-9934-fbdc3d22cafc" containerName="main" Jul 14 20:31:11.402201 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.401990 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f3338cc-96fb-4a33-9934-fbdc3d22cafc" containerName="main" Jul 14 20:31:11.402201 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.402042 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa9457c0-720a-4920-a10f-d479af6978de" containerName="main" Jul 14 20:31:11.402201 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.402051 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f3338cc-96fb-4a33-9934-fbdc3d22cafc" containerName="main" Jul 14 20:31:11.404893 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.404874 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.407845 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.407815 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a99cbd\"/\"kube-root-ca.crt\"" Jul 14 20:31:11.407988 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.407852 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a99cbd\"/\"openshift-service-ca.crt\"" Jul 14 20:31:11.407988 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.407819 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a99cbd\"/\"default-dockercfg-5fhmn\"" Jul 14 20:31:11.407988 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.407819 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a99cbd\"/\"rate-test-model-a99cbd-kserve-self-signed-certs\"" Jul 14 20:31:11.413007 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.412984 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v"] Jul 14 20:31:11.442752 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.442720 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-tmp-dir\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.442936 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.442815 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-dshm\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.442936 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.442900 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-tls-certs\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.443281 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.443244 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-model-cache\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.443430 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.443286 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76tvw\" (UniqueName: \"kubernetes.io/projected/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-kube-api-access-76tvw\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.443430 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.443325 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-home\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.543975 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.543936 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-dshm\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.543975 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.543981 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-tls-certs\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.544250 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.544012 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-model-cache\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.544250 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.544029 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76tvw\" (UniqueName: \"kubernetes.io/projected/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-kube-api-access-76tvw\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.544250 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.544049 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-home\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.544250 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.544117 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-tmp-dir\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.544561 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.544531 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-model-cache\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.544637 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.544579 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-home\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.544637 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.544592 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-tmp-dir\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.546379 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.546347 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-dshm\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.546597 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.546577 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-tls-certs\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.552412 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.552390 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76tvw\" (UniqueName: \"kubernetes.io/projected/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-kube-api-access-76tvw\") pod \"rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.714880 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.714773 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:11.845157 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:11.845128 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v"] Jul 14 20:31:11.847519 ip-10-0-132-184 kubenswrapper[2557]: W0714 20:31:11.847493 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod998a2e65_d64d_4dd2_bb74_bbc4e5ca2fc4.slice/crio-ac3b275d33d597d42179e5a45c5bcda65a805fc698ddf8e6b8aa5410be9b1972 WatchSource:0}: Error finding container ac3b275d33d597d42179e5a45c5bcda65a805fc698ddf8e6b8aa5410be9b1972: Status 404 returned error can't find the container with id ac3b275d33d597d42179e5a45c5bcda65a805fc698ddf8e6b8aa5410be9b1972 Jul 14 20:31:12.043273 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:12.043186 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" event={"ID":"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4","Type":"ContainerStarted","Data":"998b605f49eed8a35bfe6a8fb888fac7b15811bb9a41872e4704ef85de18e8d8"} Jul 14 20:31:12.043273 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:12.043226 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" event={"ID":"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4","Type":"ContainerStarted","Data":"ac3b275d33d597d42179e5a45c5bcda65a805fc698ddf8e6b8aa5410be9b1972"} Jul 14 20:31:12.064402 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:12.064349 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" podStartSLOduration=1.06433091 podStartE2EDuration="1.06433091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:31:12.063006922 +0000 UTC m=+2690.142752715" watchObservedRunningTime="2026-07-14 20:31:12.06433091 +0000 UTC m=+2690.144076702" Jul 14 20:31:16.189299 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:16.189267 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:31:21.715392 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:21.715355 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:21.716187 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:21.715669 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:21.727729 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:21.727709 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:22.099299 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:22.099273 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:31:22.482459 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:22.482378 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 20:31:22.488085 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:22.488050 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-926lj_15d09c81-aa45-4b25-83c2-c13f788e021c/ovn-acl-logging/0.log" Jul 14 20:31:50.287804 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:31:50.287768 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:32:05.096326 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:05.096287 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:32:14.178050 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:14.178013 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v"] Jul 14 20:32:15.175471 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:15.175433 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7502-140498342391856"] Jul 14 20:32:15.178879 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:15.178864 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342391856" Jul 14 20:32:15.181035 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:15.181009 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v9vdk\"" Jul 14 20:32:15.184361 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:15.184337 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7502-140498342391856"] Jul 14 20:32:15.334021 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:15.333988 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6j6s\" (UniqueName: \"kubernetes.io/projected/8526caa5-bbaa-4a0b-a36d-0db66fd60f4c-kube-api-access-f6j6s\") pod \"test-curl-7502-140498342391856\" (UID: \"8526caa5-bbaa-4a0b-a36d-0db66fd60f4c\") " pod="opendatahub/test-curl-7502-140498342391856" Jul 14 20:32:15.435200 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:15.435104 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6j6s\" (UniqueName: \"kubernetes.io/projected/8526caa5-bbaa-4a0b-a36d-0db66fd60f4c-kube-api-access-f6j6s\") pod \"test-curl-7502-140498342391856\" (UID: \"8526caa5-bbaa-4a0b-a36d-0db66fd60f4c\") " pod="opendatahub/test-curl-7502-140498342391856" Jul 14 20:32:15.445715 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:15.445686 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6j6s\" (UniqueName: \"kubernetes.io/projected/8526caa5-bbaa-4a0b-a36d-0db66fd60f4c-kube-api-access-f6j6s\") pod \"test-curl-7502-140498342391856\" (UID: \"8526caa5-bbaa-4a0b-a36d-0db66fd60f4c\") " pod="opendatahub/test-curl-7502-140498342391856" Jul 14 20:32:15.488276 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:15.488247 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342391856" Jul 14 20:32:15.617940 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:15.617917 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7502-140498342391856"] Jul 14 20:32:15.620482 ip-10-0-132-184 kubenswrapper[2557]: W0714 20:32:15.620455 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8526caa5_bbaa_4a0b_a36d_0db66fd60f4c.slice/crio-4b6a07ac0d50b54db72ea39fe89f6441816f44120bc40695f7b98fec005a171b WatchSource:0}: Error finding container 4b6a07ac0d50b54db72ea39fe89f6441816f44120bc40695f7b98fec005a171b: Status 404 returned error can't find the container with id 4b6a07ac0d50b54db72ea39fe89f6441816f44120bc40695f7b98fec005a171b Jul 14 20:32:15.621808 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:15.621788 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 20:32:16.330969 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:16.330939 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7502-140498342391856" event={"ID":"8526caa5-bbaa-4a0b-a36d-0db66fd60f4c","Type":"ContainerStarted","Data":"4b6a07ac0d50b54db72ea39fe89f6441816f44120bc40695f7b98fec005a171b"} Jul 14 20:32:17.337014 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:17.336979 2557 generic.go:356] "Generic (PLEG): container finished" podID="8526caa5-bbaa-4a0b-a36d-0db66fd60f4c" containerID="9d4f8d1f1372b72a3bd3d4d2ecebac1e3837e24289a0c1532fb7c03206a75f5c" exitCode=0 Jul 14 20:32:17.337409 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:17.337024 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7502-140498342391856" event={"ID":"8526caa5-bbaa-4a0b-a36d-0db66fd60f4c","Type":"ContainerDied","Data":"9d4f8d1f1372b72a3bd3d4d2ecebac1e3837e24289a0c1532fb7c03206a75f5c"} Jul 14 20:32:18.474661 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.474630 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342391856" Jul 14 20:32:18.515163 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.515134 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7502-140498342391856_8526caa5-bbaa-4a0b-a36d-0db66fd60f4c/test-curl-7502-140498342391856/0.log" Jul 14 20:32:18.543517 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.543485 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7502-140498342391856"] Jul 14 20:32:18.548975 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.548947 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7502-140498342391856"] Jul 14 20:32:18.664683 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.664585 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6j6s\" (UniqueName: \"kubernetes.io/projected/8526caa5-bbaa-4a0b-a36d-0db66fd60f4c-kube-api-access-f6j6s\") pod \"8526caa5-bbaa-4a0b-a36d-0db66fd60f4c\" (UID: \"8526caa5-bbaa-4a0b-a36d-0db66fd60f4c\") " Jul 14 20:32:18.666819 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.666790 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8526caa5-bbaa-4a0b-a36d-0db66fd60f4c-kube-api-access-f6j6s" (OuterVolumeSpecName: "kube-api-access-f6j6s") pod "8526caa5-bbaa-4a0b-a36d-0db66fd60f4c" (UID: "8526caa5-bbaa-4a0b-a36d-0db66fd60f4c"). InnerVolumeSpecName "kube-api-access-f6j6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:32:18.693847 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.693817 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7502-140498342395184"] Jul 14 20:32:18.694243 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.694225 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8526caa5-bbaa-4a0b-a36d-0db66fd60f4c" containerName="test-curl-7502-140498342391856" Jul 14 20:32:18.694337 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.694245 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8526caa5-bbaa-4a0b-a36d-0db66fd60f4c" containerName="test-curl-7502-140498342391856" Jul 14 20:32:18.694337 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.694331 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8526caa5-bbaa-4a0b-a36d-0db66fd60f4c" containerName="test-curl-7502-140498342391856" Jul 14 20:32:18.697323 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.697304 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342395184" Jul 14 20:32:18.700970 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.700944 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7502-140498342395184"] Jul 14 20:32:18.765352 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.765313 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6j6s\" (UniqueName: \"kubernetes.io/projected/8526caa5-bbaa-4a0b-a36d-0db66fd60f4c-kube-api-access-f6j6s\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:32:18.866567 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.866531 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkrs7\" (UniqueName: \"kubernetes.io/projected/9bf375ed-0fb0-4f68-a406-de758d64339d-kube-api-access-vkrs7\") pod \"test-curl-7502-140498342395184\" (UID: \"9bf375ed-0fb0-4f68-a406-de758d64339d\") " pod="opendatahub/test-curl-7502-140498342395184" Jul 14 20:32:18.968024 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.967955 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkrs7\" (UniqueName: \"kubernetes.io/projected/9bf375ed-0fb0-4f68-a406-de758d64339d-kube-api-access-vkrs7\") pod \"test-curl-7502-140498342395184\" (UID: \"9bf375ed-0fb0-4f68-a406-de758d64339d\") " pod="opendatahub/test-curl-7502-140498342395184" Jul 14 20:32:18.976810 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:18.976782 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkrs7\" (UniqueName: \"kubernetes.io/projected/9bf375ed-0fb0-4f68-a406-de758d64339d-kube-api-access-vkrs7\") pod \"test-curl-7502-140498342395184\" (UID: \"9bf375ed-0fb0-4f68-a406-de758d64339d\") " pod="opendatahub/test-curl-7502-140498342395184" Jul 14 20:32:19.007666 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:19.007635 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342395184" Jul 14 20:32:19.138007 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:19.137977 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7502-140498342395184"] Jul 14 20:32:19.140258 ip-10-0-132-184 kubenswrapper[2557]: W0714 20:32:19.140227 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bf375ed_0fb0_4f68_a406_de758d64339d.slice/crio-bdc2e90d81c5ff1b714baad38425a19ba0e7bd7e9d417661bc33aea626ddf408 WatchSource:0}: Error finding container bdc2e90d81c5ff1b714baad38425a19ba0e7bd7e9d417661bc33aea626ddf408: Status 404 returned error can't find the container with id bdc2e90d81c5ff1b714baad38425a19ba0e7bd7e9d417661bc33aea626ddf408 Jul 14 20:32:19.348137 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:19.348113 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342391856" Jul 14 20:32:19.348569 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:19.348164 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b6a07ac0d50b54db72ea39fe89f6441816f44120bc40695f7b98fec005a171b" Jul 14 20:32:19.349320 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:19.349298 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7502-140498342395184" event={"ID":"9bf375ed-0fb0-4f68-a406-de758d64339d","Type":"ContainerStarted","Data":"bdc2e90d81c5ff1b714baad38425a19ba0e7bd7e9d417661bc33aea626ddf408"} Jul 14 20:32:19.358051 ip-10-0-132-184 kubenswrapper[2557]: E0714 20:32:19.358025 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7502-140498342391856\" is forbidden: User \"system:node:ip-10-0-132-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-184.ec2.internal' and this object" podUID="8526caa5-bbaa-4a0b-a36d-0db66fd60f4c" pod="opendatahub/test-curl-7502-140498342391856" Jul 14 20:32:19.607717 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:19.607634 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:32:20.355413 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:20.355377 2557 generic.go:356] "Generic (PLEG): container finished" podID="9bf375ed-0fb0-4f68-a406-de758d64339d" containerID="7e5ba69583808b8a4db98734dc1cc2e661ae449e234b76b5f20e6ecf45eefe74" exitCode=0 Jul 14 20:32:20.357220 ip-10-0-132-184 kubenswrapper[2557]: E0714 20:32:20.357185 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7502-140498342391856\" is forbidden: User \"system:node:ip-10-0-132-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-184.ec2.internal' and this object" podUID="8526caa5-bbaa-4a0b-a36d-0db66fd60f4c" pod="opendatahub/test-curl-7502-140498342391856" Jul 14 20:32:20.361017 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:20.360996 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8526caa5-bbaa-4a0b-a36d-0db66fd60f4c" path="/var/lib/kubelet/pods/8526caa5-bbaa-4a0b-a36d-0db66fd60f4c/volumes" Jul 14 20:32:20.361278 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:20.361263 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7502-140498342395184" event={"ID":"9bf375ed-0fb0-4f68-a406-de758d64339d","Type":"ContainerDied","Data":"7e5ba69583808b8a4db98734dc1cc2e661ae449e234b76b5f20e6ecf45eefe74"} Jul 14 20:32:21.488300 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:21.488276 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342395184" Jul 14 20:32:21.491407 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:21.491386 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkrs7\" (UniqueName: \"kubernetes.io/projected/9bf375ed-0fb0-4f68-a406-de758d64339d-kube-api-access-vkrs7\") pod \"9bf375ed-0fb0-4f68-a406-de758d64339d\" (UID: \"9bf375ed-0fb0-4f68-a406-de758d64339d\") " Jul 14 20:32:21.493400 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:21.493370 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf375ed-0fb0-4f68-a406-de758d64339d-kube-api-access-vkrs7" (OuterVolumeSpecName: "kube-api-access-vkrs7") pod "9bf375ed-0fb0-4f68-a406-de758d64339d" (UID: "9bf375ed-0fb0-4f68-a406-de758d64339d"). InnerVolumeSpecName "kube-api-access-vkrs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:32:21.508930 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:21.508905 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7502-140498342395184_9bf375ed-0fb0-4f68-a406-de758d64339d/test-curl-7502-140498342395184/0.log" Jul 14 20:32:21.536076 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:21.535977 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7502-140498342395184"] Jul 14 20:32:21.542692 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:21.542660 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7502-140498342395184"] Jul 14 20:32:21.592241 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:21.592204 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkrs7\" (UniqueName: \"kubernetes.io/projected/9bf375ed-0fb0-4f68-a406-de758d64339d-kube-api-access-vkrs7\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:32:22.362081 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:22.362032 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bf375ed-0fb0-4f68-a406-de758d64339d" path="/var/lib/kubelet/pods/9bf375ed-0fb0-4f68-a406-de758d64339d/volumes" Jul 14 20:32:22.365857 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:22.365833 2557 scope.go:117] "RemoveContainer" containerID="7e5ba69583808b8a4db98734dc1cc2e661ae449e234b76b5f20e6ecf45eefe74" Jul 14 20:32:22.365979 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:22.365838 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342395184" Jul 14 20:32:28.025139 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:28.025106 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7502-140498342229024"] Jul 14 20:32:28.025638 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:28.025447 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bf375ed-0fb0-4f68-a406-de758d64339d" containerName="test-curl-7502-140498342395184" Jul 14 20:32:28.025638 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:28.025459 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bf375ed-0fb0-4f68-a406-de758d64339d" containerName="test-curl-7502-140498342395184" Jul 14 20:32:28.025638 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:28.025517 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bf375ed-0fb0-4f68-a406-de758d64339d" containerName="test-curl-7502-140498342395184" Jul 14 20:32:28.029680 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:28.029663 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342229024" Jul 14 20:32:28.031697 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:28.031673 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v9vdk\"" Jul 14 20:32:28.034938 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:28.034914 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7502-140498342229024"] Jul 14 20:32:28.048594 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:28.048571 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r8jq\" (UniqueName: \"kubernetes.io/projected/301c15ad-d223-4166-bb18-ae4f87ff3062-kube-api-access-8r8jq\") pod \"test-curl-7502-140498342229024\" (UID: \"301c15ad-d223-4166-bb18-ae4f87ff3062\") " pod="opendatahub/test-curl-7502-140498342229024" Jul 14 20:32:28.149372 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:28.149336 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r8jq\" (UniqueName: \"kubernetes.io/projected/301c15ad-d223-4166-bb18-ae4f87ff3062-kube-api-access-8r8jq\") pod \"test-curl-7502-140498342229024\" (UID: \"301c15ad-d223-4166-bb18-ae4f87ff3062\") " pod="opendatahub/test-curl-7502-140498342229024" Jul 14 20:32:28.157403 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:28.157380 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r8jq\" (UniqueName: \"kubernetes.io/projected/301c15ad-d223-4166-bb18-ae4f87ff3062-kube-api-access-8r8jq\") pod \"test-curl-7502-140498342229024\" (UID: \"301c15ad-d223-4166-bb18-ae4f87ff3062\") " pod="opendatahub/test-curl-7502-140498342229024" Jul 14 20:32:28.339946 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:28.339923 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342229024" Jul 14 20:32:28.676852 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:28.676772 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7502-140498342229024"] Jul 14 20:32:28.680200 ip-10-0-132-184 kubenswrapper[2557]: W0714 20:32:28.680173 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod301c15ad_d223_4166_bb18_ae4f87ff3062.slice/crio-06f99f195d6ccb427b109d821ad822b2cff306fdc728c18b69d03a846f2e7e07 WatchSource:0}: Error finding container 06f99f195d6ccb427b109d821ad822b2cff306fdc728c18b69d03a846f2e7e07: Status 404 returned error can't find the container with id 06f99f195d6ccb427b109d821ad822b2cff306fdc728c18b69d03a846f2e7e07 Jul 14 20:32:29.199354 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.199309 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" podUID="998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4" containerName="main" containerID="cri-o://998b605f49eed8a35bfe6a8fb888fac7b15811bb9a41872e4704ef85de18e8d8" gracePeriod=45 Jul 14 20:32:29.399906 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.399877 2557 generic.go:356] "Generic (PLEG): container finished" podID="301c15ad-d223-4166-bb18-ae4f87ff3062" containerID="22fcad67ff1797e2de442872bb0ec89ad6107050331dd369a66d879b98e7bf67" exitCode=0 Jul 14 20:32:29.400088 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.399970 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7502-140498342229024" event={"ID":"301c15ad-d223-4166-bb18-ae4f87ff3062","Type":"ContainerDied","Data":"22fcad67ff1797e2de442872bb0ec89ad6107050331dd369a66d879b98e7bf67"} Jul 14 20:32:29.400088 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.400017 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7502-140498342229024" event={"ID":"301c15ad-d223-4166-bb18-ae4f87ff3062","Type":"ContainerStarted","Data":"06f99f195d6ccb427b109d821ad822b2cff306fdc728c18b69d03a846f2e7e07"} Jul 14 20:32:29.401563 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.401542 2557 generic.go:356] "Generic (PLEG): container finished" podID="998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4" containerID="998b605f49eed8a35bfe6a8fb888fac7b15811bb9a41872e4704ef85de18e8d8" exitCode=0 Jul 14 20:32:29.401670 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.401579 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" event={"ID":"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4","Type":"ContainerDied","Data":"998b605f49eed8a35bfe6a8fb888fac7b15811bb9a41872e4704ef85de18e8d8"} Jul 14 20:32:29.441312 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.441287 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:32:29.563897 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.563860 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76tvw\" (UniqueName: \"kubernetes.io/projected/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-kube-api-access-76tvw\") pod \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " Jul 14 20:32:29.564108 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.563924 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-home\") pod \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " Jul 14 20:32:29.564108 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.563964 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-tmp-dir\") pod \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " Jul 14 20:32:29.564108 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.564006 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-dshm\") pod \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " Jul 14 20:32:29.564108 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.564095 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-tls-certs\") pod \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " Jul 14 20:32:29.564349 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.564123 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-model-cache\") pod \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\" (UID: \"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4\") " Jul 14 20:32:29.564349 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.564225 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-home" (OuterVolumeSpecName: "home") pod "998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4" (UID: "998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:32:29.564349 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.564244 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4" (UID: "998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:32:29.564505 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.564433 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-model-cache" (OuterVolumeSpecName: "model-cache") pod "998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4" (UID: "998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:32:29.564505 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.564441 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-home\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:32:29.564505 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.564472 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-tmp-dir\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:32:29.566113 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.566035 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-dshm" (OuterVolumeSpecName: "dshm") pod "998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4" (UID: "998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:32:29.566113 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.566100 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-kube-api-access-76tvw" (OuterVolumeSpecName: "kube-api-access-76tvw") pod "998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4" (UID: "998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4"). InnerVolumeSpecName "kube-api-access-76tvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:32:29.566245 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.566110 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4" (UID: "998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:32:29.665899 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.665862 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-dshm\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:32:29.665899 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.665892 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-tls-certs\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:32:29.665899 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.665902 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-model-cache\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:32:29.666156 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:29.665912 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76tvw\" (UniqueName: \"kubernetes.io/projected/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4-kube-api-access-76tvw\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:32:30.406890 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.406860 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" Jul 14 20:32:30.407376 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.406910 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v" event={"ID":"998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4","Type":"ContainerDied","Data":"ac3b275d33d597d42179e5a45c5bcda65a805fc698ddf8e6b8aa5410be9b1972"} Jul 14 20:32:30.407376 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.406967 2557 scope.go:117] "RemoveContainer" containerID="998b605f49eed8a35bfe6a8fb888fac7b15811bb9a41872e4704ef85de18e8d8" Jul 14 20:32:30.424212 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.424183 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v"] Jul 14 20:32:30.430659 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.430635 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/rate-test-model-a99cbd-kserve-5b5566d4d7-lwq4v"] Jul 14 20:32:30.541228 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.541206 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342229024" Jul 14 20:32:30.562071 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.562027 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7502-140498342229024_301c15ad-d223-4166-bb18-ae4f87ff3062/test-curl-7502-140498342229024/0.log" Jul 14 20:32:30.573306 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.573283 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r8jq\" (UniqueName: \"kubernetes.io/projected/301c15ad-d223-4166-bb18-ae4f87ff3062-kube-api-access-8r8jq\") pod \"301c15ad-d223-4166-bb18-ae4f87ff3062\" (UID: \"301c15ad-d223-4166-bb18-ae4f87ff3062\") " Jul 14 20:32:30.575337 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.575303 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301c15ad-d223-4166-bb18-ae4f87ff3062-kube-api-access-8r8jq" (OuterVolumeSpecName: "kube-api-access-8r8jq") pod "301c15ad-d223-4166-bb18-ae4f87ff3062" (UID: "301c15ad-d223-4166-bb18-ae4f87ff3062"). InnerVolumeSpecName "kube-api-access-8r8jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:32:30.585090 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.585036 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7502-140498342229024"] Jul 14 20:32:30.587398 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.587375 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7502-140498342229024"] Jul 14 20:32:30.674507 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.674425 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r8jq\" (UniqueName: \"kubernetes.io/projected/301c15ad-d223-4166-bb18-ae4f87ff3062-kube-api-access-8r8jq\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:32:30.741711 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.741679 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7502-140498342229024"] Jul 14 20:32:30.742085 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.742049 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="301c15ad-d223-4166-bb18-ae4f87ff3062" containerName="test-curl-7502-140498342229024" Jul 14 20:32:30.742181 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.742087 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="301c15ad-d223-4166-bb18-ae4f87ff3062" containerName="test-curl-7502-140498342229024" Jul 14 20:32:30.742181 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.742114 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4" containerName="main" Jul 14 20:32:30.742181 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.742123 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4" containerName="main" Jul 14 20:32:30.742333 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.742222 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="301c15ad-d223-4166-bb18-ae4f87ff3062" containerName="test-curl-7502-140498342229024" Jul 14 20:32:30.742333 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.742239 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4" containerName="main" Jul 14 20:32:30.745152 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.745131 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342229024" Jul 14 20:32:30.750186 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.750160 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7502-140498342229024"] Jul 14 20:32:30.775073 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.775022 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9zc5\" (UniqueName: \"kubernetes.io/projected/a5e04fc4-0a5d-4e69-a203-b65fda09c1d5-kube-api-access-h9zc5\") pod \"test-curl-7502-140498342229024\" (UID: \"a5e04fc4-0a5d-4e69-a203-b65fda09c1d5\") " pod="opendatahub/test-curl-7502-140498342229024" Jul 14 20:32:30.875767 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.875725 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9zc5\" (UniqueName: \"kubernetes.io/projected/a5e04fc4-0a5d-4e69-a203-b65fda09c1d5-kube-api-access-h9zc5\") pod \"test-curl-7502-140498342229024\" (UID: \"a5e04fc4-0a5d-4e69-a203-b65fda09c1d5\") " pod="opendatahub/test-curl-7502-140498342229024" Jul 14 20:32:30.884961 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:30.884937 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9zc5\" (UniqueName: \"kubernetes.io/projected/a5e04fc4-0a5d-4e69-a203-b65fda09c1d5-kube-api-access-h9zc5\") pod \"test-curl-7502-140498342229024\" (UID: \"a5e04fc4-0a5d-4e69-a203-b65fda09c1d5\") " pod="opendatahub/test-curl-7502-140498342229024" Jul 14 20:32:31.055136 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:31.055042 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342229024" Jul 14 20:32:31.389889 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:31.389862 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7502-140498342229024"] Jul 14 20:32:31.391755 ip-10-0-132-184 kubenswrapper[2557]: W0714 20:32:31.391727 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5e04fc4_0a5d_4e69_a203_b65fda09c1d5.slice/crio-2dffde68249c3fe251aec78d6fa0c1ffb1eb196ad774850dc2ecd5286e219a27 WatchSource:0}: Error finding container 2dffde68249c3fe251aec78d6fa0c1ffb1eb196ad774850dc2ecd5286e219a27: Status 404 returned error can't find the container with id 2dffde68249c3fe251aec78d6fa0c1ffb1eb196ad774850dc2ecd5286e219a27 Jul 14 20:32:31.413342 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:31.413309 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7502-140498342229024" event={"ID":"a5e04fc4-0a5d-4e69-a203-b65fda09c1d5","Type":"ContainerStarted","Data":"2dffde68249c3fe251aec78d6fa0c1ffb1eb196ad774850dc2ecd5286e219a27"} Jul 14 20:32:31.414669 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:31.414647 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342229024" Jul 14 20:32:31.414758 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:31.414655 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06f99f195d6ccb427b109d821ad822b2cff306fdc728c18b69d03a846f2e7e07" Jul 14 20:32:31.426725 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:31.426692 2557 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7502-140498342229024" oldPodUID="301c15ad-d223-4166-bb18-ae4f87ff3062" podUID="a5e04fc4-0a5d-4e69-a203-b65fda09c1d5" Jul 14 20:32:32.362581 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:32.362545 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="301c15ad-d223-4166-bb18-ae4f87ff3062" path="/var/lib/kubelet/pods/301c15ad-d223-4166-bb18-ae4f87ff3062/volumes" Jul 14 20:32:32.362945 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:32.362932 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4" path="/var/lib/kubelet/pods/998a2e65-d64d-4dd2-bb74-bbc4e5ca2fc4/volumes" Jul 14 20:32:32.421224 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:32.421187 2557 generic.go:356] "Generic (PLEG): container finished" podID="a5e04fc4-0a5d-4e69-a203-b65fda09c1d5" containerID="9dd559a81885796784890372a97128aee860778c266af58951b239a2a1dba370" exitCode=0 Jul 14 20:32:32.421663 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:32.421248 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7502-140498342229024" event={"ID":"a5e04fc4-0a5d-4e69-a203-b65fda09c1d5","Type":"ContainerDied","Data":"9dd559a81885796784890372a97128aee860778c266af58951b239a2a1dba370"} Jul 14 20:32:33.558138 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:33.558109 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342229024" Jul 14 20:32:33.579503 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:33.579481 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7502-140498342229024_a5e04fc4-0a5d-4e69-a203-b65fda09c1d5/test-curl-7502-140498342229024/0.log" Jul 14 20:32:33.595550 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:33.595510 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9zc5\" (UniqueName: \"kubernetes.io/projected/a5e04fc4-0a5d-4e69-a203-b65fda09c1d5-kube-api-access-h9zc5\") pod \"a5e04fc4-0a5d-4e69-a203-b65fda09c1d5\" (UID: \"a5e04fc4-0a5d-4e69-a203-b65fda09c1d5\") " Jul 14 20:32:33.597559 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:33.597523 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e04fc4-0a5d-4e69-a203-b65fda09c1d5-kube-api-access-h9zc5" (OuterVolumeSpecName: "kube-api-access-h9zc5") pod "a5e04fc4-0a5d-4e69-a203-b65fda09c1d5" (UID: "a5e04fc4-0a5d-4e69-a203-b65fda09c1d5"). InnerVolumeSpecName "kube-api-access-h9zc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:32:33.601863 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:33.601836 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7502-140498342229024"] Jul 14 20:32:33.606707 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:33.606685 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7502-140498342229024"] Jul 14 20:32:33.696604 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:33.696516 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9zc5\" (UniqueName: \"kubernetes.io/projected/a5e04fc4-0a5d-4e69-a203-b65fda09c1d5-kube-api-access-h9zc5\") on node \"ip-10-0-132-184.ec2.internal\" DevicePath \"\"" Jul 14 20:32:34.361259 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:34.361227 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5e04fc4-0a5d-4e69-a203-b65fda09c1d5" path="/var/lib/kubelet/pods/a5e04fc4-0a5d-4e69-a203-b65fda09c1d5/volumes" Jul 14 20:32:34.432892 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:34.432866 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342229024" Jul 14 20:32:34.433069 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:34.432867 2557 scope.go:117] "RemoveContainer" containerID="9dd559a81885796784890372a97128aee860778c266af58951b239a2a1dba370" Jul 14 20:32:47.595200 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:32:47.595113 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:33:14.005292 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:14.005258 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8nkkl"] Jul 14 20:33:22.582969 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:22.582939 2557 scope.go:117] "RemoveContainer" containerID="db0a6d0bc9da8baf8cf5d6723928615c11c2e0c11dfbcd3e478c793ffccb6412" Jul 14 20:33:26.481688 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:26.481653 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-76b84bf67b-4qxqp_c455ffb7-c1b9-412c-bd91-4c104eef9b5c/authorino/0.log" Jul 14 20:33:32.017508 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:32.017476 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f59db9768-652mp_cccba306-e43d-4a4b-be0d-9b4a462f463e/copy-manifests/0.log" Jul 14 20:33:32.030792 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:32.030764 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f59db9768-652mp_cccba306-e43d-4a4b-be0d-9b4a462f463e/manager/0.log" Jul 14 20:33:35.864011 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:35.863978 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-76b84bf67b-4qxqp_c455ffb7-c1b9-412c-bd91-4c104eef9b5c/authorino/0.log" Jul 14 20:33:36.589212 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:36.589183 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8nkkl_7bd8be7c-548c-4bc7-9d2d-69cf3457ee82/limitador/0.log" Jul 14 20:33:36.713874 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:36.713842 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-74x5n_95f534a0-4a84-42c0-ac9b-3f7b354d4a1b/manager/0.log" Jul 14 20:33:37.469122 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:37.469097 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5df66fb8c-s42jf_5eef7830-07e1-4a2c-9734-fb9d39eb4c13/kube-auth-proxy/0.log" Jul 14 20:33:38.687999 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:38.687968 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbdz66v_63bda19c-fc1e-4c20-9f5e-6c0234759068/main/0.log" Jul 14 20:33:38.931326 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:38.931292 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-ldwlf_ce6d5f17-7fc2-4535-988f-5dd5fcb74c41/main/0.log" Jul 14 20:33:46.474465 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:46.474429 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6w8gx_d3f090a3-021a-4185-bcc8-7c00d7d56b77/global-pull-secret-syncer/0.log" Jul 14 20:33:46.645886 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:46.645856 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ff7s5_322d4ffe-4783-4a5a-a18f-47878e117000/konnectivity-agent/0.log" Jul 14 20:33:46.667244 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:46.667219 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-184.ec2.internal_130523dbb878b2a1af0eb57e85edc42c/haproxy/0.log" Jul 14 20:33:52.012509 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:52.012477 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-76b84bf67b-4qxqp_c455ffb7-c1b9-412c-bd91-4c104eef9b5c/authorino/0.log" Jul 14 20:33:52.566929 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:52.566894 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8nkkl_7bd8be7c-548c-4bc7-9d2d-69cf3457ee82/limitador/0.log" Jul 14 20:33:52.630892 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:52.630863 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-74x5n_95f534a0-4a84-42c0-ac9b-3f7b354d4a1b/manager/0.log" Jul 14 20:33:54.561632 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:54.561594 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-cqtfv_0b73d9ec-522c-4b17-ab16-b21fdae87b75/kube-state-metrics/0.log" Jul 14 20:33:54.582504 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:54.582476 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-cqtfv_0b73d9ec-522c-4b17-ab16-b21fdae87b75/kube-rbac-proxy-main/0.log" Jul 14 20:33:54.606193 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:54.606119 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-cqtfv_0b73d9ec-522c-4b17-ab16-b21fdae87b75/kube-rbac-proxy-self/0.log" Jul 14 20:33:54.702785 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:54.702757 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4sth_222df6f4-91ed-4e2f-842d-ddf0d767a6d8/node-exporter/0.log" Jul 14 20:33:54.734702 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:54.734675 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4sth_222df6f4-91ed-4e2f-842d-ddf0d767a6d8/kube-rbac-proxy/0.log" Jul 14 20:33:54.756594 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:54.756572 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c4sth_222df6f4-91ed-4e2f-842d-ddf0d767a6d8/init-textfile/0.log" Jul 14 20:33:55.097719 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:55.097691 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec172125-9bd4-4684-9a94-34e9e4f4af50/prometheus/0.log" Jul 14 20:33:55.123376 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:55.123348 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec172125-9bd4-4684-9a94-34e9e4f4af50/config-reloader/0.log" Jul 14 20:33:55.146157 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:55.146129 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec172125-9bd4-4684-9a94-34e9e4f4af50/thanos-sidecar/0.log" Jul 14 20:33:55.168636 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:55.168609 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec172125-9bd4-4684-9a94-34e9e4f4af50/kube-rbac-proxy-web/0.log" Jul 14 20:33:55.191651 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:55.191624 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec172125-9bd4-4684-9a94-34e9e4f4af50/kube-rbac-proxy/0.log" Jul 14 20:33:55.217953 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:55.217923 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec172125-9bd4-4684-9a94-34e9e4f4af50/kube-rbac-proxy-thanos/0.log" Jul 14 20:33:55.242641 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:55.242616 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ec172125-9bd4-4684-9a94-34e9e4f4af50/init-config-reloader/0.log" Jul 14 20:33:55.272006 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:55.271978 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-zzxjn_0d559014-bde8-4125-b1d5-7ae7f113c52b/prometheus-operator/0.log" Jul 14 20:33:55.292114 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:55.292086 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-zzxjn_0d559014-bde8-4125-b1d5-7ae7f113c52b/kube-rbac-proxy/0.log" Jul 14 20:33:55.435684 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:55.435607 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65889bfdcf-qzcqn_2aca72f4-54b5-4676-96eb-78ebd8f25877/thanos-query/0.log" Jul 14 20:33:55.457664 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:55.457634 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65889bfdcf-qzcqn_2aca72f4-54b5-4676-96eb-78ebd8f25877/kube-rbac-proxy-web/0.log" Jul 14 20:33:55.479778 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:55.479742 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65889bfdcf-qzcqn_2aca72f4-54b5-4676-96eb-78ebd8f25877/kube-rbac-proxy/0.log" Jul 14 20:33:55.501118 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:55.501088 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65889bfdcf-qzcqn_2aca72f4-54b5-4676-96eb-78ebd8f25877/prom-label-proxy/0.log" Jul 14 20:33:55.525727 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:55.525701 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65889bfdcf-qzcqn_2aca72f4-54b5-4676-96eb-78ebd8f25877/kube-rbac-proxy-rules/0.log" Jul 14 20:33:55.548121 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:55.548094 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65889bfdcf-qzcqn_2aca72f4-54b5-4676-96eb-78ebd8f25877/kube-rbac-proxy-metrics/0.log" Jul 14 20:33:56.439027 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.438994 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs"] Jul 14 20:33:56.439436 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.439395 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5e04fc4-0a5d-4e69-a203-b65fda09c1d5" containerName="test-curl-7502-140498342229024" Jul 14 20:33:56.439436 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.439417 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5e04fc4-0a5d-4e69-a203-b65fda09c1d5" containerName="test-curl-7502-140498342229024" Jul 14 20:33:56.439607 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.439524 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5e04fc4-0a5d-4e69-a203-b65fda09c1d5" containerName="test-curl-7502-140498342229024" Jul 14 20:33:56.442918 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.442897 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.445007 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.444986 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7v52j\"/\"openshift-service-ca.crt\"" Jul 14 20:33:56.445139 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.444990 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7v52j\"/\"default-dockercfg-tbpgw\"" Jul 14 20:33:56.445757 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.445741 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7v52j\"/\"kube-root-ca.crt\"" Jul 14 20:33:56.452147 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.452121 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs"] Jul 14 20:33:56.516699 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.516664 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c7e20ea-a54c-46c5-9589-bb564572f107-lib-modules\") pod \"perf-node-gather-daemonset-2hvqs\" (UID: \"9c7e20ea-a54c-46c5-9589-bb564572f107\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.516699 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.516701 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c7e20ea-a54c-46c5-9589-bb564572f107-sys\") pod \"perf-node-gather-daemonset-2hvqs\" (UID: \"9c7e20ea-a54c-46c5-9589-bb564572f107\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.516907 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.516765 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c7e20ea-a54c-46c5-9589-bb564572f107-proc\") pod \"perf-node-gather-daemonset-2hvqs\" (UID: \"9c7e20ea-a54c-46c5-9589-bb564572f107\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.516907 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.516825 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c7e20ea-a54c-46c5-9589-bb564572f107-podres\") pod \"perf-node-gather-daemonset-2hvqs\" (UID: \"9c7e20ea-a54c-46c5-9589-bb564572f107\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.516907 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.516871 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr5l2\" (UniqueName: \"kubernetes.io/projected/9c7e20ea-a54c-46c5-9589-bb564572f107-kube-api-access-pr5l2\") pod \"perf-node-gather-daemonset-2hvqs\" (UID: \"9c7e20ea-a54c-46c5-9589-bb564572f107\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.617970 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.617934 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c7e20ea-a54c-46c5-9589-bb564572f107-lib-modules\") pod \"perf-node-gather-daemonset-2hvqs\" (UID: \"9c7e20ea-a54c-46c5-9589-bb564572f107\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.617970 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.617970 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c7e20ea-a54c-46c5-9589-bb564572f107-sys\") pod \"perf-node-gather-daemonset-2hvqs\" (UID: \"9c7e20ea-a54c-46c5-9589-bb564572f107\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.618258 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.617995 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c7e20ea-a54c-46c5-9589-bb564572f107-proc\") pod \"perf-node-gather-daemonset-2hvqs\" (UID: \"9c7e20ea-a54c-46c5-9589-bb564572f107\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.618258 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.618021 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c7e20ea-a54c-46c5-9589-bb564572f107-podres\") pod \"perf-node-gather-daemonset-2hvqs\" (UID: \"9c7e20ea-a54c-46c5-9589-bb564572f107\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.618258 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.618087 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr5l2\" (UniqueName: \"kubernetes.io/projected/9c7e20ea-a54c-46c5-9589-bb564572f107-kube-api-access-pr5l2\") pod \"perf-node-gather-daemonset-2hvqs\" (UID: \"9c7e20ea-a54c-46c5-9589-bb564572f107\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.618258 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.618104 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c7e20ea-a54c-46c5-9589-bb564572f107-proc\") pod \"perf-node-gather-daemonset-2hvqs\" (UID: \"9c7e20ea-a54c-46c5-9589-bb564572f107\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.618258 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.618103 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c7e20ea-a54c-46c5-9589-bb564572f107-sys\") pod \"perf-node-gather-daemonset-2hvqs\" (UID: \"9c7e20ea-a54c-46c5-9589-bb564572f107\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.618258 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.618149 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c7e20ea-a54c-46c5-9589-bb564572f107-lib-modules\") pod \"perf-node-gather-daemonset-2hvqs\" (UID: \"9c7e20ea-a54c-46c5-9589-bb564572f107\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.618258 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.618224 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c7e20ea-a54c-46c5-9589-bb564572f107-podres\") pod \"perf-node-gather-daemonset-2hvqs\" (UID: \"9c7e20ea-a54c-46c5-9589-bb564572f107\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.627960 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.627940 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr5l2\" (UniqueName: \"kubernetes.io/projected/9c7e20ea-a54c-46c5-9589-bb564572f107-kube-api-access-pr5l2\") pod \"perf-node-gather-daemonset-2hvqs\" (UID: \"9c7e20ea-a54c-46c5-9589-bb564572f107\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.754588 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.754488 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:56.888499 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:56.888476 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs"] Jul 14 20:33:56.890994 ip-10-0-132-184 kubenswrapper[2557]: W0714 20:33:56.890968 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9c7e20ea_a54c_46c5_9589_bb564572f107.slice/crio-bd85909759684dce329bb5b5e2ec8fc16ea42e5efd4562dfe84c8b5f661f238a WatchSource:0}: Error finding container bd85909759684dce329bb5b5e2ec8fc16ea42e5efd4562dfe84c8b5f661f238a: Status 404 returned error can't find the container with id bd85909759684dce329bb5b5e2ec8fc16ea42e5efd4562dfe84c8b5f661f238a Jul 14 20:33:57.757740 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:57.757710 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7fbf959699-lg2wp_e70ccdd9-a7a2-441d-931f-38c86f0ad341/console/0.log" Jul 14 20:33:57.815992 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:57.815959 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" event={"ID":"9c7e20ea-a54c-46c5-9589-bb564572f107","Type":"ContainerStarted","Data":"3a583dd53acc03322ec3f044c17c944adcbfa774e0d47a84391671bd85a8a98c"} Jul 14 20:33:57.816176 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:57.816009 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" event={"ID":"9c7e20ea-a54c-46c5-9589-bb564572f107","Type":"ContainerStarted","Data":"bd85909759684dce329bb5b5e2ec8fc16ea42e5efd4562dfe84c8b5f661f238a"} Jul 14 20:33:57.816176 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:57.816094 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:33:57.832434 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:57.832386 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" podStartSLOduration=1.832370637 podStartE2EDuration="1.832370637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 20:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:33:57.831410963 +0000 UTC m=+2855.911156777" watchObservedRunningTime="2026-07-14 20:33:57.832370637 +0000 UTC m=+2855.912116428" Jul 14 20:33:59.395582 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:59.395555 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9fb96_06b84d98-4dc1-47d4-8330-b292e14940f1/dns/0.log" Jul 14 20:33:59.423139 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:59.423114 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9fb96_06b84d98-4dc1-47d4-8330-b292e14940f1/kube-rbac-proxy/0.log" Jul 14 20:33:59.552273 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:33:59.552242 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lpvwr_fd7db434-46fc-4a27-83b2-acaefc0bd727/dns-node-resolver/0.log" Jul 14 20:34:00.211945 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:34:00.211917 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-knrrs_839de40a-5fee-44af-8173-a65990f845cf/node-ca/0.log" Jul 14 20:34:01.406223 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:34:01.406194 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5df66fb8c-s42jf_5eef7830-07e1-4a2c-9734-fb9d39eb4c13/kube-auth-proxy/0.log" Jul 14 20:34:02.285601 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:34:02.285568 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-49ddf_4a7dce1a-0e21-4d14-a65e-2e2b28d69f70/serve-healthcheck-canary/0.log" Jul 14 20:34:02.929896 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:34:02.929867 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-krtpl_af5e2ec9-a131-4164-b80e-3045f199612b/kube-rbac-proxy/0.log" Jul 14 20:34:02.953991 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:34:02.953966 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-krtpl_af5e2ec9-a131-4164-b80e-3045f199612b/exporter/0.log" Jul 14 20:34:02.976722 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:34:02.976700 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-krtpl_af5e2ec9-a131-4164-b80e-3045f199612b/extractor/0.log" Jul 14 20:34:03.830866 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:34:03.830838 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-2hvqs" Jul 14 20:34:05.880769 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:34:05.880739 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f59db9768-652mp_cccba306-e43d-4a4b-be0d-9b4a462f463e/manager/0.log" Jul 14 20:34:05.899554 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:34:05.899527 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f59db9768-652mp_cccba306-e43d-4a4b-be0d-9b4a462f463e/copy-manifests/0.log" Jul 14 20:34:06.837757 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:34:06.837732 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5df66fb8c-s42jf_5eef7830-07e1-4a2c-9734-fb9d39eb4c13/kube-auth-proxy/0.log" Jul 14 20:34:07.301962 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:34:07.301884 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5dc78d48f4-2ft4x_9f6db25a-56f4-4636-b6e4-56c351a834c9/manager/0.log" Jul 14 20:34:08.420391 ip-10-0-132-184 kubenswrapper[2557]: I0714 20:34:08.420362 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5df66fb8c-s42jf_5eef7830-07e1-4a2c-9734-fb9d39eb4c13/kube-auth-proxy/0.log"