Jul 22 09:15:47.285541 ip-10-0-134-55 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 09:15:47.285701 ip-10-0-134-55 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 09:15:47.285714 ip-10-0-134-55 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 09:15:47.286016 ip-10-0-134-55 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 09:15:57.299884 ip-10-0-134-55 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 09:15:57.299904 ip-10-0-134-55 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1cbf9d89dfea4eb7856828000e946a54 -- Jul 22 09:17:55.384255 ip-10-0-134-55 systemd[1]: Starting Kubernetes Kubelet... Jul 22 09:17:55.777300 ip-10-0-134-55 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 09:17:55.777300 ip-10-0-134-55 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 09:17:55.777300 ip-10-0-134-55 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 09:17:55.777300 ip-10-0-134-55 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 09:17:55.777300 ip-10-0-134-55 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 09:17:55.777956 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.777355 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 09:17:55.783724 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783697 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 09:17:55.783724 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783714 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 09:17:55.783724 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783720 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 09:17:55.783724 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783723 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 09:17:55.783724 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783726 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 09:17:55.783724 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783729 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 09:17:55.783724 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783732 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783738 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783742 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783745 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783748 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783751 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783754 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783756 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783759 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783762 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783765 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783768 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783771 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783773 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783776 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783778 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783781 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783783 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783791 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 09:17:55.784063 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783795 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783798 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783800 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783803 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783806 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783809 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783813 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783816 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783818 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783821 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783824 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783826 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783829 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783831 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783834 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783837 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783840 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783843 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783846 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783849 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 09:17:55.784501 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783851 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783854 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783857 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783859 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783862 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783864 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783867 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783869 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783872 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783876 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783879 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783882 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783885 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783887 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783890 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783892 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783895 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783897 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783900 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783903 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 09:17:55.784998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783907 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783910 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783914 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783917 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783919 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783923 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783926 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783929 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783944 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783947 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783950 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783953 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783955 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783958 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783960 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783963 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783966 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783969 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783972 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783975 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 09:17:55.785468 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783977 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783980 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783983 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783985 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783988 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783990 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783993 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783995 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.783998 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784000 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784003 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784006 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784010 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784013 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784735 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784741 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784744 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784747 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784750 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 09:17:55.785998 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784753 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784758 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784762 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784765 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784768 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784771 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784773 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784776 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784779 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784782 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784784 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784787 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784790 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784793 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784795 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784798 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784801 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784803 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784805 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784808 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 09:17:55.786455 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784811 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784813 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784815 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784818 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784820 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784823 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784826 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784828 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784831 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784833 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784836 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784840 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784843 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784846 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784849 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784851 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784855 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784859 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784862 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 09:17:55.786956 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784864 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784868 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784872 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784875 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784879 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784881 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784884 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784887 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784890 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784892 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784895 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784898 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784901 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784903 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784905 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784908 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784911 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784913 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784915 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784918 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 09:17:55.787416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784920 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784923 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784925 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784928 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784946 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784950 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784953 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784955 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784958 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784961 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784963 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784966 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784969 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784971 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784974 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784976 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784979 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784981 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784984 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784987 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 09:17:55.787886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784989 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784992 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784995 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.784997 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785000 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785003 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785006 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785008 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785011 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785013 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785016 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785019 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785021 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785024 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785026 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785500 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785506 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785510 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785514 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785517 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 09:17:55.788380 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785519 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785522 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785525 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785527 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785530 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785532 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785535 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785537 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785541 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785545 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785548 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785551 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785554 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785556 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785559 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785562 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785564 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785567 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785569 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 09:17:55.788866 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785572 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785574 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785577 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785579 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785582 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785584 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785587 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785590 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785592 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785595 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785598 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785600 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785604 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785607 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785610 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785614 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785617 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785619 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785622 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 09:17:55.789344 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785624 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785626 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785629 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785631 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785634 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785636 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785640 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785644 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785647 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785649 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785652 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785656 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785659 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785661 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785664 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785666 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785669 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785671 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785674 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785677 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 09:17:55.789783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785679 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785681 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785684 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785687 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785689 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785692 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785694 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785697 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785699 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785701 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785704 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785706 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785709 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785711 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785714 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785716 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785719 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785723 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785726 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785729 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 09:17:55.790282 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785732 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785734 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785736 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785739 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785741 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785744 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785746 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785749 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785751 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785754 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785756 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785758 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785761 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785763 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785766 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.785769 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785833 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785845 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785850 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785855 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785859 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 09:17:55.790783 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785862 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785866 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785870 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785873 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785876 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785879 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785882 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785885 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785889 2564 flags.go:64] FLAG: --cgroup-root="" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785892 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785895 2564 flags.go:64] FLAG: --client-ca-file="" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785899 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785902 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785907 2564 flags.go:64] FLAG: --cluster-domain="" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785909 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785913 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785915 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785919 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785922 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785925 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785928 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785945 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785948 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785951 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 09:17:55.791351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785953 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785956 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785962 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785966 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785969 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785972 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785974 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785977 2564 flags.go:64] FLAG: --enable-server="true" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785980 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785984 2564 flags.go:64] FLAG: --event-burst="100" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785988 2564 flags.go:64] FLAG: --event-qps="50" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785990 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785993 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.785996 2564 flags.go:64] FLAG: --eviction-hard="" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786000 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786003 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786007 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786010 2564 flags.go:64] FLAG: --eviction-soft="" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786013 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786016 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786020 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786023 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786026 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786028 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786031 2564 flags.go:64] FLAG: --feature-gates="" Jul 22 09:17:55.791910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786035 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786038 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786041 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786044 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786047 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786050 2564 flags.go:64] FLAG: --help="false" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786052 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-134-55.ec2.internal" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786055 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786058 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786063 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786066 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786070 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786073 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786075 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786078 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786081 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786084 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786087 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786090 2564 flags.go:64] FLAG: --kube-reserved="" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786093 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786096 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786099 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786101 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786105 2564 flags.go:64] FLAG: --lock-file="" Jul 22 09:17:55.792554 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786108 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786111 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786114 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786119 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786123 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786125 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786128 2564 flags.go:64] FLAG: --logging-format="text" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786131 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786134 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786138 2564 flags.go:64] FLAG: --manifest-url="" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786140 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786145 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786147 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786151 2564 flags.go:64] FLAG: --max-pods="110" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786154 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786157 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786160 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786164 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786167 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786170 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786173 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786181 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786184 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786187 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 09:17:55.793555 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786190 2564 flags.go:64] FLAG: --pod-cidr="" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786193 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786198 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786201 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786204 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786207 2564 flags.go:64] FLAG: --port="10250" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786210 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786214 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00f3b4a64098c99c0" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786217 2564 flags.go:64] FLAG: --qos-reserved="" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786220 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786223 2564 flags.go:64] FLAG: --register-node="true" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786226 2564 flags.go:64] FLAG: --register-with-taints="" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786230 2564 flags.go:64] FLAG: --registry-burst="10" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786243 2564 flags.go:64] FLAG: --registry-qps="5" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786247 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786250 2564 flags.go:64] FLAG: --reserved-memory="" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786257 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786259 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786262 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786265 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786268 2564 flags.go:64] FLAG: --runonce="false" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786271 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786274 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786277 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786280 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 09:17:55.794573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786283 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786287 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786290 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786293 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786295 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786298 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786301 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786304 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786307 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786310 2564 flags.go:64] FLAG: --system-cgroups="" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786313 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786318 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786321 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786323 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786329 2564 flags.go:64] FLAG: --tls-min-version="" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786332 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786335 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786338 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786340 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786344 2564 flags.go:64] FLAG: --v="2" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786347 2564 flags.go:64] FLAG: --version="false" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786352 2564 flags.go:64] FLAG: --vmodule="" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786355 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.786358 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786482 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 09:17:55.795726 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786486 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786489 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786493 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786496 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786498 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786501 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786504 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786506 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786510 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786513 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786516 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786518 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786521 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786523 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786526 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786529 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786531 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786534 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786536 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786539 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 09:17:55.796728 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786542 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786547 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786550 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786553 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786556 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786559 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786562 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786564 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786567 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786570 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786572 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786575 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786578 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786580 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786583 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786585 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786588 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786590 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786593 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 09:17:55.797365 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786596 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786600 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786602 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786605 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786608 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786611 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786615 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786618 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786620 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786623 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786625 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786628 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786630 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786633 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786637 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786640 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786644 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786647 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786650 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 09:17:55.798156 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786653 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786655 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786658 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786661 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786664 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786666 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786669 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786671 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786674 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786676 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786679 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786682 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786684 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786687 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786691 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786693 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786696 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786698 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786701 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786704 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 09:17:55.798837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786706 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786708 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786711 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786714 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786716 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786719 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786721 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786726 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786728 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786731 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786733 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786736 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786738 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786741 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786743 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786746 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786748 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786752 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786755 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 09:17:55.799517 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.786757 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 09:17:55.800038 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.787381 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 09:17:55.800038 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.795012 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 09:17:55.800038 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.795031 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 09:17:55.800038 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.795058 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 09:17:55.800038 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.795069 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 09:17:55.800038 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795170 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 09:17:55.800038 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795178 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 09:17:55.800038 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795183 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 09:17:55.800038 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795188 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 09:17:55.800038 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795192 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 09:17:55.800038 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795196 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 09:17:55.800038 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795201 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 09:17:55.800038 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795205 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 09:17:55.800038 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795209 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795214 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795219 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795223 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795227 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795231 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795236 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795240 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795244 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795249 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795253 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795258 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795262 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795266 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795270 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795274 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795278 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795283 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795287 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795292 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 09:17:55.800389 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795296 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795302 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795306 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795311 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795315 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795319 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795323 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795328 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795332 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795336 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795341 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795345 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795349 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795353 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795357 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795362 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795366 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795371 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795375 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795379 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 09:17:55.800919 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795384 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795388 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795392 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795396 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795400 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795404 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795409 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795413 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795417 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795421 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795425 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795429 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795434 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795439 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795444 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795451 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795456 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795460 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795465 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795469 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 09:17:55.801416 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795473 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795478 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795482 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795487 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795493 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795499 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795503 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795508 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795512 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795517 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795521 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795525 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795529 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795534 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795538 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795542 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795546 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795551 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795555 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795562 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 09:17:55.801973 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795567 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 09:17:55.802541 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795573 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 09:17:55.802541 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795579 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 09:17:55.802541 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795583 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 09:17:55.802541 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795588 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 09:17:55.802541 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795593 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 09:17:55.802541 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795597 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 09:17:55.802541 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795608 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 09:17:55.802541 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795613 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 09:17:55.802541 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795618 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 09:17:55.802541 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795623 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 09:17:55.802541 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.795631 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 09:17:55.802541 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795841 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 09:17:55.802541 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795850 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 09:17:55.802541 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795855 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 09:17:55.802541 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795859 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795865 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795872 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795876 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795880 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795885 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795889 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795894 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795898 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795903 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795907 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795911 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795915 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795920 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795924 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795929 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795951 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795956 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795960 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795964 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 09:17:55.803057 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795968 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795973 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795978 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795982 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795987 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795992 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.795997 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796001 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796005 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796010 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796014 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796018 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796024 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796030 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796035 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796039 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796043 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796048 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796052 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796056 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 09:17:55.803860 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796060 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796064 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796069 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796073 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796078 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796082 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796086 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796090 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796095 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796099 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796103 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796106 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796111 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796115 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796120 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796124 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796127 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796132 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796136 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796140 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 09:17:55.804690 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796144 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796148 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796153 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796156 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796161 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796165 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796169 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796173 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796177 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796181 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796185 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796189 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796196 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796202 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796206 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796211 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796216 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796220 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796225 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 09:17:55.805520 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796229 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796234 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796238 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796242 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796246 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796250 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796255 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796259 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796263 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796267 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796271 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796275 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796281 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796285 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796289 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796294 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 09:17:55.806198 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:55.796297 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 09:17:55.806598 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.796306 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 09:17:55.806598 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.796507 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 09:17:55.806598 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.799131 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 09:17:55.806598 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.800767 2564 server.go:1019] "Starting client certificate rotation" Jul 22 09:17:55.806598 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.800884 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 09:17:55.806598 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.800923 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 09:17:55.806598 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.803601 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 09:17:55.806598 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.805951 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 09:17:55.815377 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.815361 2564 log.go:25] "Validated CRI v1 runtime API" Jul 22 09:17:55.817222 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.817207 2564 log.go:25] "Validated CRI v1 image API" Jul 22 09:17:55.818013 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.817999 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 09:17:55.820137 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.820120 2564 fs.go:135] Filesystem UUIDs: map[29ca1058-a038-4140-8a22-985641ccd9d6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 925e8c9b-5d8c-4d21-97a2-20806f868e36:/dev/nvme0n1p3] Jul 22 09:17:55.820199 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.820136 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 09:17:55.824559 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.824464 2564 manager.go:216] Machine: {Timestamp:2026-07-22 09:17:55.823430535 +0000 UTC m=+0.320937959 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101934 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f9186d2a29f9e7d46629d350e9481 SystemUUID:ec2f9186-d2a2-9f9e-7d46-629d350e9481 BootID:1cbf9d89-dfea-4eb7-8568-28000e946a54 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:13:36:21:23:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:13:36:21:23:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:9d:5a:48:3d:9c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 09:17:55.824559 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.824552 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 09:17:55.824666 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.824621 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 09:17:55.825486 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.825465 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 09:17:55.825613 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.825489 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-55.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 09:17:55.826473 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.826463 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 09:17:55.826504 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.826475 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 09:17:55.826504 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.826481 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 09:17:55.827017 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.827004 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 09:17:55.828042 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.828032 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 09:17:55.831173 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.831164 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 22 09:17:55.831290 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.831281 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 09:17:55.831884 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.831876 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 22 09:17:55.831912 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.831888 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 09:17:55.831912 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.831897 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 09:17:55.831912 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.831905 2564 kubelet.go:398] "Adding apiserver pod source" Jul 22 09:17:55.832034 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.831913 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 09:17:55.832265 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.832254 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 09:17:55.832313 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.832271 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 09:17:55.834121 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.834104 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 09:17:55.835540 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.835519 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 09:17:55.835966 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.835952 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 09:17:55.836005 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.835995 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 09:17:55.836983 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.836971 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 09:17:55.837013 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.836989 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 09:17:55.837013 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.836995 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 09:17:55.837013 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.837001 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 09:17:55.837013 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.837010 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 09:17:55.837100 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.837015 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 09:17:55.837100 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.837022 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 09:17:55.837100 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.837027 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 09:17:55.837100 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.837033 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 09:17:55.837100 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.837038 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 09:17:55.837100 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.837045 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 09:17:55.837100 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.837051 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 09:17:55.837631 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.837622 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 09:17:55.837631 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.837630 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 09:17:55.840686 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.840675 2564 server.go:1268] "Started kubelet" Jul 22 09:17:55.840752 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.840730 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 09:17:55.840808 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.840753 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 09:17:55.840889 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.840817 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 09:17:55.841475 ip-10-0-134-55 systemd[1]: Started Kubernetes Kubelet. Jul 22 09:17:55.842448 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.842323 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 09:17:55.842925 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.842913 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 22 09:17:55.843675 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:55.843643 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 09:17:55.844032 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:55.844000 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-55.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 09:17:55.844125 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.843739 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-55.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 09:17:55.846491 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:55.846476 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 09:17:55.848246 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.848231 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 09:17:55.848320 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.848280 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 09:17:55.848376 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:55.848352 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-55.ec2.internal\" not found" Jul 22 09:17:55.848799 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.848782 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 09:17:55.848799 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.848775 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 09:17:55.848926 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.848804 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 09:17:55.848926 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.848901 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 22 09:17:55.848926 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.848907 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 22 09:17:55.849068 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.848995 2564 factory.go:55] Registering systemd factory Jul 22 09:17:55.849068 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.849011 2564 factory.go:223] Registration of the systemd container factory successfully Jul 22 09:17:55.849189 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.849176 2564 factory.go:153] Registering CRI-O factory Jul 22 09:17:55.849189 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.849189 2564 factory.go:223] Registration of the crio container factory successfully Jul 22 09:17:55.849283 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.849247 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 09:17:55.849283 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.849272 2564 factory.go:103] Registering Raw factory Jul 22 09:17:55.849283 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.849285 2564 manager.go:1196] Started watching for new ooms in manager Jul 22 09:17:55.849736 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:55.848661 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-55.ec2.internal.18c491fec9e732b5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-55.ec2.internal,UID:ip-10-0-134-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-55.ec2.internal,},FirstTimestamp:2026-07-22 09:17:55.840651957 +0000 UTC m=+0.338159381,LastTimestamp:2026-07-22 09:17:55.840651957 +0000 UTC m=+0.338159381,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-55.ec2.internal,}" Jul 22 09:17:55.850072 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.850056 2564 manager.go:349] Starting recovery of all containers Jul 22 09:17:55.855078 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:55.855051 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 09:17:55.855187 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:55.855144 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-55.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 09:17:55.859309 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.859295 2564 manager.go:354] Recovery completed Jul 22 09:17:55.860669 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.860649 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2rqxk" Jul 22 09:17:55.863137 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.863126 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 09:17:55.864694 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.864682 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasSufficientMemory" Jul 22 09:17:55.864738 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.864706 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 09:17:55.864738 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.864719 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasSufficientPID" Jul 22 09:17:55.865195 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.865182 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 09:17:55.865195 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.865192 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 09:17:55.865195 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.865195 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 09:17:55.865347 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.865211 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 22 09:17:55.865729 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.865718 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasSufficientMemory" Jul 22 09:17:55.865765 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.865735 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 09:17:55.865765 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.865742 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasSufficientPID" Jul 22 09:17:55.866360 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.866349 2564 policy_none.go:49] "None policy: Start" Jul 22 09:17:55.866424 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.866366 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 09:17:55.866424 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.866378 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 09:17:55.866570 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:55.866513 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-55.ec2.internal.18c491fecb560e15 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-55.ec2.internal,UID:ip-10-0-134-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-55.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-55.ec2.internal,},FirstTimestamp:2026-07-22 09:17:55.864694293 +0000 UTC m=+0.362201719,LastTimestamp:2026-07-22 09:17:55.864694293 +0000 UTC m=+0.362201719,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-55.ec2.internal,}" Jul 22 09:17:55.867365 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.867355 2564 policy_none.go:47] "Start" Jul 22 09:17:55.870336 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.870319 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2rqxk" Jul 22 09:17:55.876518 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:55.876465 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-55.ec2.internal.18c491fecb565b29 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-55.ec2.internal,UID:ip-10-0-134-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-55.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-55.ec2.internal,},FirstTimestamp:2026-07-22 09:17:55.864714025 +0000 UTC m=+0.362221449,LastTimestamp:2026-07-22 09:17:55.864714025 +0000 UTC m=+0.362221449,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-55.ec2.internal,}" Jul 22 09:17:55.910512 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.910410 2564 manager.go:337] "Starting Device Plugin manager" Jul 22 09:17:55.910586 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:55.910540 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 09:17:55.910586 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.910548 2564 server.go:87] "Starting device plugin registration server" Jul 22 09:17:55.910769 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.910759 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 09:17:55.910808 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.910770 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 09:17:55.910882 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.910865 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 09:17:55.911005 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.910971 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 09:17:55.911005 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:55.910978 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 09:17:55.911391 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:55.911362 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 09:17:55.911448 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:55.911418 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-55.ec2.internal\" not found" Jul 22 09:17:56.006816 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.006789 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 09:17:56.008001 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.007980 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 09:17:56.008066 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.008024 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 09:17:56.008066 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.008045 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 09:17:56.008146 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:56.008075 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 09:17:56.009950 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.009912 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 09:17:56.011334 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.011323 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 09:17:56.012032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.012017 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasSufficientMemory" Jul 22 09:17:56.012092 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.012047 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 09:17:56.012092 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.012059 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasSufficientPID" Jul 22 09:17:56.012092 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.012079 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.019063 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.019050 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.019103 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:56.019073 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-55.ec2.internal\": node \"ip-10-0-134-55.ec2.internal\" not found" Jul 22 09:17:56.031089 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:56.031048 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-55.ec2.internal\" not found" Jul 22 09:17:56.108488 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.108457 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-55.ec2.internal"] Jul 22 09:17:56.108631 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.108553 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 09:17:56.109964 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.109950 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasSufficientMemory" Jul 22 09:17:56.110051 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.109979 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 09:17:56.110051 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.109992 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasSufficientPID" Jul 22 09:17:56.111059 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.111048 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 09:17:56.111186 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.111173 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.111270 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.111205 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 09:17:56.111701 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.111683 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasSufficientMemory" Jul 22 09:17:56.111783 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.111707 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 09:17:56.111783 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.111717 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasSufficientPID" Jul 22 09:17:56.111783 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.111748 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasSufficientMemory" Jul 22 09:17:56.111783 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.111770 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 09:17:56.111783 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.111781 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasSufficientPID" Jul 22 09:17:56.112736 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.112725 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.112788 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.112747 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 09:17:56.113325 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.113313 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasSufficientMemory" Jul 22 09:17:56.113392 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.113334 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 09:17:56.113392 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.113347 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeHasSufficientPID" Jul 22 09:17:56.131757 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:56.131738 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-55.ec2.internal\" not found" Jul 22 09:17:56.137366 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:56.137350 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-55.ec2.internal\" not found" node="ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.141324 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:56.141310 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-55.ec2.internal\" not found" node="ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.150403 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.150384 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e0fc6bf224341812e0ad602c5f4c985-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal\" (UID: \"7e0fc6bf224341812e0ad602c5f4c985\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.150479 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.150407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e0fc6bf224341812e0ad602c5f4c985-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal\" (UID: \"7e0fc6bf224341812e0ad602c5f4c985\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.150479 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.150426 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fceedcd76d463c4089e6465fad28b1d6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-55.ec2.internal\" (UID: \"fceedcd76d463c4089e6465fad28b1d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.232006 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:56.231973 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-55.ec2.internal\" not found" Jul 22 09:17:56.250871 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.250852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fceedcd76d463c4089e6465fad28b1d6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-55.ec2.internal\" (UID: \"fceedcd76d463c4089e6465fad28b1d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.250962 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.250875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e0fc6bf224341812e0ad602c5f4c985-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal\" (UID: \"7e0fc6bf224341812e0ad602c5f4c985\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.250962 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.250890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e0fc6bf224341812e0ad602c5f4c985-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal\" (UID: \"7e0fc6bf224341812e0ad602c5f4c985\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.250962 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.250953 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fceedcd76d463c4089e6465fad28b1d6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-55.ec2.internal\" (UID: \"fceedcd76d463c4089e6465fad28b1d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.251051 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.250968 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7e0fc6bf224341812e0ad602c5f4c985-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal\" (UID: \"7e0fc6bf224341812e0ad602c5f4c985\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.251051 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.250953 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e0fc6bf224341812e0ad602c5f4c985-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal\" (UID: \"7e0fc6bf224341812e0ad602c5f4c985\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.332611 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:56.332564 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-55.ec2.internal\" not found" Jul 22 09:17:56.433030 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:56.432998 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-55.ec2.internal\" not found" Jul 22 09:17:56.440109 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.440092 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.443973 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.443536 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-55.ec2.internal" Jul 22 09:17:56.533372 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:56.533351 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-55.ec2.internal\" not found" Jul 22 09:17:56.633761 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:56.633717 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-55.ec2.internal\" not found" Jul 22 09:17:56.671516 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.671495 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 09:17:56.734293 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:56.734269 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-55.ec2.internal\" not found" Jul 22 09:17:56.799657 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.799636 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 09:17:56.800035 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.799771 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 09:17:56.800035 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.799792 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 09:17:56.835036 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:56.835012 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-55.ec2.internal\" not found" Jul 22 09:17:56.873480 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.873451 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 09:12:55 +0000 UTC" deadline="2028-04-23 17:14:36.803187814 +0000 UTC" Jul 22 09:17:56.873480 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.873476 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15391h56m39.929714002s" Jul 22 09:17:56.925179 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:56.925153 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfceedcd76d463c4089e6465fad28b1d6.slice/crio-9f0d49f5bf1090303988ea82d1aa3a88a63cea56a030f81467eb56d76ec98ae3 WatchSource:0}: Error finding container 9f0d49f5bf1090303988ea82d1aa3a88a63cea56a030f81467eb56d76ec98ae3: Status 404 returned error can't find the container with id 9f0d49f5bf1090303988ea82d1aa3a88a63cea56a030f81467eb56d76ec98ae3 Jul 22 09:17:56.930879 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:56.930865 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 09:17:56.935413 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:56.935399 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-55.ec2.internal\" not found" Jul 22 09:17:57.011027 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.010984 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-55.ec2.internal" event={"ID":"fceedcd76d463c4089e6465fad28b1d6","Type":"ContainerStarted","Data":"9f0d49f5bf1090303988ea82d1aa3a88a63cea56a030f81467eb56d76ec98ae3"} Jul 22 09:17:57.036143 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:57.036118 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-55.ec2.internal\" not found" Jul 22 09:17:57.047065 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.047045 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 09:17:57.049074 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.049057 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal" Jul 22 09:17:57.063441 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.063423 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 09:17:57.065151 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.065139 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-55.ec2.internal" Jul 22 09:17:57.075079 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.075061 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 09:17:57.083468 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.083449 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 09:17:57.833842 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.833812 2564 apiserver.go:52] "Watching apiserver" Jul 22 09:17:57.844701 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.844677 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 09:17:57.845049 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.845025 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-llddr","kube-system/konnectivity-agent-xqb5q","openshift-dns/node-resolver-p9fv9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal","openshift-multus/network-metrics-daemon-2wqk7","openshift-network-diagnostics/network-check-target-km7cq","openshift-network-operator/iptables-alerter-phzcz","openshift-ovn-kubernetes/ovnkube-node-mdr8h","kube-system/global-pull-secret-syncer-5zw6v","kube-system/kube-apiserver-proxy-ip-10-0-134-55.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k","openshift-cluster-node-tuning-operator/tuned-n48m7","openshift-image-registry/node-ca-98sq4","openshift-multus/multus-additional-cni-plugins-x8j5l"] Jul 22 09:17:57.846953 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.846923 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:17:57.847028 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:57.847005 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:17:57.848010 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.847992 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-phzcz" Jul 22 09:17:57.848882 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.848865 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 09:17:57.848990 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.848971 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-98sq4" Jul 22 09:17:57.850026 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.850005 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xqb5q" Jul 22 09:17:57.851195 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.851177 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p9fv9" Jul 22 09:17:57.851315 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.851299 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xfkrf\"" Jul 22 09:17:57.851393 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.851328 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 09:17:57.851835 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.851819 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 09:17:57.851923 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.851850 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 09:17:57.852254 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.852239 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-llddr" Jul 22 09:17:57.853298 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.853283 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:17:57.853378 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:57.853351 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:17:57.853512 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.853488 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 09:17:57.853615 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.853575 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 09:17:57.853615 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.853583 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 09:17:57.854664 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.854546 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.854664 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.854545 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4sh8g\"" Jul 22 09:17:57.855337 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.855320 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j8kjf\"" Jul 22 09:17:57.855612 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.855597 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:17:57.855684 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:57.855651 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:17:57.856786 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.856769 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9425446d-1c7a-419d-8020-59828616976a-agent-certs\") pod \"konnectivity-agent-xqb5q\" (UID: \"9425446d-1c7a-419d-8020-59828616976a\") " pod="kube-system/konnectivity-agent-xqb5q" Jul 22 09:17:57.856869 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.856791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9425446d-1c7a-419d-8020-59828616976a-konnectivity-ca\") pod \"konnectivity-agent-xqb5q\" (UID: \"9425446d-1c7a-419d-8020-59828616976a\") " pod="kube-system/konnectivity-agent-xqb5q" Jul 22 09:17:57.856869 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.856809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74lfl\" (UniqueName: \"kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl\") pod \"network-check-target-km7cq\" (UID: \"ec80e86d-c05c-439a-ad34-c44e4009f060\") " pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:17:57.856869 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.856844 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c007a031-f135-4b00-8075-b9e362cb09f5-iptables-alerter-script\") pod \"iptables-alerter-phzcz\" (UID: \"c007a031-f135-4b00-8075-b9e362cb09f5\") " pod="openshift-network-operator/iptables-alerter-phzcz" Jul 22 09:17:57.856869 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.856850 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:57.857081 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.856881 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c007a031-f135-4b00-8075-b9e362cb09f5-host-slash\") pod \"iptables-alerter-phzcz\" (UID: \"c007a031-f135-4b00-8075-b9e362cb09f5\") " pod="openshift-network-operator/iptables-alerter-phzcz" Jul 22 09:17:57.857081 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.856926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkd9s\" (UniqueName: \"kubernetes.io/projected/c007a031-f135-4b00-8075-b9e362cb09f5-kube-api-access-lkd9s\") pod \"iptables-alerter-phzcz\" (UID: \"c007a031-f135-4b00-8075-b9e362cb09f5\") " pod="openshift-network-operator/iptables-alerter-phzcz" Jul 22 09:17:57.857081 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.857016 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d4fc604-5bcb-4274-a20d-5710d075dce3-host\") pod \"node-ca-98sq4\" (UID: \"9d4fc604-5bcb-4274-a20d-5710d075dce3\") " pod="openshift-image-registry/node-ca-98sq4" Jul 22 09:17:57.857081 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.857041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d4fc604-5bcb-4274-a20d-5710d075dce3-serviceca\") pod \"node-ca-98sq4\" (UID: \"9d4fc604-5bcb-4274-a20d-5710d075dce3\") " pod="openshift-image-registry/node-ca-98sq4" Jul 22 09:17:57.857232 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.857064 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjcnc\" (UniqueName: \"kubernetes.io/projected/9d4fc604-5bcb-4274-a20d-5710d075dce3-kube-api-access-tjcnc\") pod \"node-ca-98sq4\" (UID: \"9d4fc604-5bcb-4274-a20d-5710d075dce3\") " pod="openshift-image-registry/node-ca-98sq4" Jul 22 09:17:57.858095 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.858081 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:57.858502 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.858437 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 09:17:57.858502 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.858450 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 09:17:57.858502 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.858458 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 09:17:57.858697 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.858586 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 09:17:57.858827 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.858786 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 09:17:57.858827 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.858806 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 09:17:57.858827 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.858815 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 09:17:57.859138 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.859124 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 09:17:57.859588 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.859570 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 09:17:57.859675 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.859628 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 09:17:57.859675 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.859647 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sh5rp\"" Jul 22 09:17:57.859777 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.859628 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 09:17:57.859777 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.859740 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 09:17:57.859777 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.859577 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 09:17:57.859777 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.859754 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 09:17:57.859963 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.859787 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x867g\"" Jul 22 09:17:57.859963 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.859863 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 09:17:57.860040 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.859977 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 09:17:57.860091 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.860077 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:57.862736 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.862723 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b9jtv\"" Jul 22 09:17:57.863252 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.863237 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 09:17:57.863252 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.863252 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 09:17:57.863418 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.863298 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5mxcj\"" Jul 22 09:17:57.863418 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.863360 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 09:17:57.863418 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.863389 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9rvmm\"" Jul 22 09:17:57.866831 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.866813 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 09:17:57.866924 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.866853 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 09:17:57.866924 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.866892 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f8nvw\"" Jul 22 09:17:57.873090 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.871016 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 09:17:57.895472 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.895456 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nv46q" Jul 22 09:17:57.904661 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.904645 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nv46q" Jul 22 09:17:57.928370 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.928350 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 09:17:57.950212 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.950192 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 09:17:57.957565 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-run-ovn\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.957639 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957567 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-sys\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:57.957639 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957583 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-etc-openvswitch\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.957639 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-dbus\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:17:57.957639 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957616 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c007a031-f135-4b00-8075-b9e362cb09f5-host-slash\") pod \"iptables-alerter-phzcz\" (UID: \"c007a031-f135-4b00-8075-b9e362cb09f5\") " pod="openshift-network-operator/iptables-alerter-phzcz" Jul 22 09:17:57.957639 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957630 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-sysctl-d\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:57.957843 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957644 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-tmp\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:57.957843 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957690 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c007a031-f135-4b00-8075-b9e362cb09f5-host-slash\") pod \"iptables-alerter-phzcz\" (UID: \"c007a031-f135-4b00-8075-b9e362cb09f5\") " pod="openshift-network-operator/iptables-alerter-phzcz" Jul 22 09:17:57.957843 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957696 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-cni-bin\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.957843 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957738 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-registration-dir\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:57.957843 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-cnibin\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.957843 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957777 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0433a177-3a7d-48d8-97b6-f48375cceb33-cni-binary-copy\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.957843 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-run-netns\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.957843 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957805 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0433a177-3a7d-48d8-97b6-f48375cceb33-multus-daemon-config\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.957843 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957820 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs\") pod \"network-metrics-daemon-2wqk7\" (UID: \"e460fb8e-afe6-4628-b08e-514c648282c3\") " pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:17:57.957843 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957843 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90be7f2f-8c43-40b8-9126-11e4f0f72f96-env-overrides\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.958226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957867 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dce67161-8f6a-4e0d-86ab-62245da6f838-cni-binary-copy\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:57.958226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-sysconfig\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:57.958226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957897 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-system-cni-dir\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.958226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957910 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-multus-conf-dir\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.958226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.957979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-kubelet-config\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:17:57.958226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dce67161-8f6a-4e0d-86ab-62245da6f838-cnibin\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:57.958226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958060 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94e33bbc-9879-4500-abd0-a3cde0baf87f-hosts-file\") pod \"node-resolver-p9fv9\" (UID: \"94e33bbc-9879-4500-abd0-a3cde0baf87f\") " pod="openshift-dns/node-resolver-p9fv9" Jul 22 09:17:57.958226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958084 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-run\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:57.958226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958103 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-multus-cni-dir\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.958226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958123 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-os-release\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.958226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958136 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-run-netns\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.958226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958153 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9425446d-1c7a-419d-8020-59828616976a-agent-certs\") pod \"konnectivity-agent-xqb5q\" (UID: \"9425446d-1c7a-419d-8020-59828616976a\") " pod="kube-system/konnectivity-agent-xqb5q" Jul 22 09:17:57.958226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74lfl\" (UniqueName: \"kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl\") pod \"network-check-target-km7cq\" (UID: \"ec80e86d-c05c-439a-ad34-c44e4009f060\") " pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c007a031-f135-4b00-8075-b9e362cb09f5-iptables-alerter-script\") pod \"iptables-alerter-phzcz\" (UID: \"c007a031-f135-4b00-8075-b9e362cb09f5\") " pod="openshift-network-operator/iptables-alerter-phzcz" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-socket-dir\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958279 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-var-lib-kubelet\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958301 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-etc-kubernetes\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958323 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dce67161-8f6a-4e0d-86ab-62245da6f838-system-cni-dir\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958381 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d4fc604-5bcb-4274-a20d-5710d075dce3-host\") pod \"node-ca-98sq4\" (UID: \"9d4fc604-5bcb-4274-a20d-5710d075dce3\") " pod="openshift-image-registry/node-ca-98sq4" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958412 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjcnc\" (UniqueName: \"kubernetes.io/projected/9d4fc604-5bcb-4274-a20d-5710d075dce3-kube-api-access-tjcnc\") pod \"node-ca-98sq4\" (UID: \"9d4fc604-5bcb-4274-a20d-5710d075dce3\") " pod="openshift-image-registry/node-ca-98sq4" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958455 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-hostroot\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958478 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d4fc604-5bcb-4274-a20d-5710d075dce3-host\") pod \"node-ca-98sq4\" (UID: \"9d4fc604-5bcb-4274-a20d-5710d075dce3\") " pod="openshift-image-registry/node-ca-98sq4" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-etc-selinux\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958494 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx92l\" (UniqueName: \"kubernetes.io/projected/21b82000-0ecd-486b-9e01-b98cf640590b-kube-api-access-kx92l\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-modprobe-d\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958579 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-multus-socket-dir-parent\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958601 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrq8n\" (UniqueName: \"kubernetes.io/projected/e460fb8e-afe6-4628-b08e-514c648282c3-kube-api-access-zrq8n\") pod \"network-metrics-daemon-2wqk7\" (UID: \"e460fb8e-afe6-4628-b08e-514c648282c3\") " pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:17:57.958784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958620 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-node-log\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958640 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dce67161-8f6a-4e0d-86ab-62245da6f838-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d4fc604-5bcb-4274-a20d-5710d075dce3-serviceca\") pod \"node-ca-98sq4\" (UID: \"9d4fc604-5bcb-4274-a20d-5710d075dce3\") " pod="openshift-image-registry/node-ca-98sq4" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-run-systemd\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958715 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90be7f2f-8c43-40b8-9126-11e4f0f72f96-ovn-node-metrics-cert\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958732 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dce67161-8f6a-4e0d-86ab-62245da6f838-os-release\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958735 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c007a031-f135-4b00-8075-b9e362cb09f5-iptables-alerter-script\") pod \"iptables-alerter-phzcz\" (UID: \"c007a031-f135-4b00-8075-b9e362cb09f5\") " pod="openshift-network-operator/iptables-alerter-phzcz" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958759 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-sys-fs\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-lib-modules\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958804 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-tuned\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-log-socket\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958854 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958896 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dce67161-8f6a-4e0d-86ab-62245da6f838-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958925 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e33bbc-9879-4500-abd0-a3cde0baf87f-tmp-dir\") pod \"node-resolver-p9fv9\" (UID: \"94e33bbc-9879-4500-abd0-a3cde0baf87f\") " pod="openshift-dns/node-resolver-p9fv9" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.958979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959002 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbsn\" (UniqueName: \"kubernetes.io/projected/0433a177-3a7d-48d8-97b6-f48375cceb33-kube-api-access-rhbsn\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.959382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959026 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959051 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90be7f2f-8c43-40b8-9126-11e4f0f72f96-ovnkube-script-lib\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8xr\" (UniqueName: \"kubernetes.io/projected/90be7f2f-8c43-40b8-9126-11e4f0f72f96-kube-api-access-nr8xr\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959104 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdmm7\" (UniqueName: \"kubernetes.io/projected/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-kube-api-access-fdmm7\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959108 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d4fc604-5bcb-4274-a20d-5710d075dce3-serviceca\") pod \"node-ca-98sq4\" (UID: \"9d4fc604-5bcb-4274-a20d-5710d075dce3\") " pod="openshift-image-registry/node-ca-98sq4" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-kubelet\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dce67161-8f6a-4e0d-86ab-62245da6f838-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959162 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5rs2\" (UniqueName: \"kubernetes.io/projected/dce67161-8f6a-4e0d-86ab-62245da6f838-kube-api-access-x5rs2\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959179 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-device-dir\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-systemd\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-var-lib-cni-bin\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959235 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-systemd-units\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959252 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-slash\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959267 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90be7f2f-8c43-40b8-9126-11e4f0f72f96-ovnkube-config\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkd9s\" (UniqueName: \"kubernetes.io/projected/c007a031-f135-4b00-8075-b9e362cb09f5-kube-api-access-lkd9s\") pod \"iptables-alerter-phzcz\" (UID: \"c007a031-f135-4b00-8075-b9e362cb09f5\") " pod="openshift-network-operator/iptables-alerter-phzcz" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-kubernetes\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:57.960032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959332 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-sysctl-conf\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:57.960485 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-var-lib-cni-multus\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.960485 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959387 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-var-lib-openvswitch\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.960485 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-cni-netd\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.960485 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959439 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gplq\" (UniqueName: \"kubernetes.io/projected/94e33bbc-9879-4500-abd0-a3cde0baf87f-kube-api-access-2gplq\") pod \"node-resolver-p9fv9\" (UID: \"94e33bbc-9879-4500-abd0-a3cde0baf87f\") " pod="openshift-dns/node-resolver-p9fv9" Jul 22 09:17:57.960485 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-host\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:57.960485 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959492 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-run-multus-certs\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.960485 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9425446d-1c7a-419d-8020-59828616976a-konnectivity-ca\") pod \"konnectivity-agent-xqb5q\" (UID: \"9425446d-1c7a-419d-8020-59828616976a\") " pod="kube-system/konnectivity-agent-xqb5q" Jul 22 09:17:57.960485 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959542 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-run-k8s-cni-cncf-io\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.960485 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-var-lib-kubelet\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:57.960485 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959594 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-run-openvswitch\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:57.960485 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.959911 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9425446d-1c7a-419d-8020-59828616976a-konnectivity-ca\") pod \"konnectivity-agent-xqb5q\" (UID: \"9425446d-1c7a-419d-8020-59828616976a\") " pod="kube-system/konnectivity-agent-xqb5q" Jul 22 09:17:57.961879 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.961857 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9425446d-1c7a-419d-8020-59828616976a-agent-certs\") pod \"konnectivity-agent-xqb5q\" (UID: \"9425446d-1c7a-419d-8020-59828616976a\") " pod="kube-system/konnectivity-agent-xqb5q" Jul 22 09:17:57.965503 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:57.965483 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 09:17:57.965503 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:57.965505 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 09:17:57.965632 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:57.965531 2564 projected.go:196] Error preparing data for projected volume kube-api-access-74lfl for pod openshift-network-diagnostics/network-check-target-km7cq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:17:57.965632 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:57.965585 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl podName:ec80e86d-c05c-439a-ad34-c44e4009f060 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:58.465565644 +0000 UTC m=+2.963073059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-74lfl" (UniqueName: "kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl") pod "network-check-target-km7cq" (UID: "ec80e86d-c05c-439a-ad34-c44e4009f060") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:17:57.967383 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.967360 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjcnc\" (UniqueName: \"kubernetes.io/projected/9d4fc604-5bcb-4274-a20d-5710d075dce3-kube-api-access-tjcnc\") pod \"node-ca-98sq4\" (UID: \"9d4fc604-5bcb-4274-a20d-5710d075dce3\") " pod="openshift-image-registry/node-ca-98sq4" Jul 22 09:17:57.967786 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:57.967769 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkd9s\" (UniqueName: \"kubernetes.io/projected/c007a031-f135-4b00-8075-b9e362cb09f5-kube-api-access-lkd9s\") pod \"iptables-alerter-phzcz\" (UID: \"c007a031-f135-4b00-8075-b9e362cb09f5\") " pod="openshift-network-operator/iptables-alerter-phzcz" Jul 22 09:17:58.060432 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-sysctl-d\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.060432 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060433 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-tmp\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.060657 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060447 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-cni-bin\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.060657 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060462 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-registration-dir\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:58.060657 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-cnibin\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.060657 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060499 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0433a177-3a7d-48d8-97b6-f48375cceb33-cni-binary-copy\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.060657 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-run-netns\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.060657 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060531 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-cni-bin\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.060657 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060563 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-registration-dir\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:58.060657 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060588 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-cnibin\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.060657 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060621 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-sysctl-d\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.060657 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0433a177-3a7d-48d8-97b6-f48375cceb33-multus-daemon-config\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060697 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs\") pod \"network-metrics-daemon-2wqk7\" (UID: \"e460fb8e-afe6-4628-b08e-514c648282c3\") " pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90be7f2f-8c43-40b8-9126-11e4f0f72f96-env-overrides\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dce67161-8f6a-4e0d-86ab-62245da6f838-cni-binary-copy\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:58.060814 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060826 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-sysconfig\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-system-cni-dir\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-multus-conf-dir\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060912 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-kubelet-config\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060952 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dce67161-8f6a-4e0d-86ab-62245da6f838-cnibin\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.060979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94e33bbc-9879-4500-abd0-a3cde0baf87f-hosts-file\") pod \"node-resolver-p9fv9\" (UID: \"94e33bbc-9879-4500-abd0-a3cde0baf87f\") " pod="openshift-dns/node-resolver-p9fv9" Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061006 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-run\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061031 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-multus-cni-dir\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061052 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-os-release\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061049 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-run-netns\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061078 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-run-netns\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.061105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061113 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dce67161-8f6a-4e0d-86ab-62245da6f838-cnibin\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061118 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-socket-dir\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-var-lib-kubelet\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061160 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-sysconfig\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-etc-kubernetes\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061193 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061206 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-system-cni-dir\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dce67161-8f6a-4e0d-86ab-62245da6f838-system-cni-dir\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061239 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-multus-conf-dir\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061243 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-hostroot\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-etc-selinux\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061279 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0433a177-3a7d-48d8-97b6-f48375cceb33-multus-daemon-config\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx92l\" (UniqueName: \"kubernetes.io/projected/21b82000-0ecd-486b-9e01-b98cf640590b-kube-api-access-kx92l\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061284 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-kubelet-config\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-modprobe-d\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061357 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-multus-socket-dir-parent\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061359 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dce67161-8f6a-4e0d-86ab-62245da6f838-system-cni-dir\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.061856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrq8n\" (UniqueName: \"kubernetes.io/projected/e460fb8e-afe6-4628-b08e-514c648282c3-kube-api-access-zrq8n\") pod \"network-metrics-daemon-2wqk7\" (UID: \"e460fb8e-afe6-4628-b08e-514c648282c3\") " pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061388 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-socket-dir\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061408 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-node-log\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061411 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-var-lib-kubelet\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061445 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dce67161-8f6a-4e0d-86ab-62245da6f838-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061476 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0433a177-3a7d-48d8-97b6-f48375cceb33-cni-binary-copy\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061492 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-run-systemd\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061382 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/90be7f2f-8c43-40b8-9126-11e4f0f72f96-env-overrides\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061504 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-etc-selinux\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90be7f2f-8c43-40b8-9126-11e4f0f72f96-ovn-node-metrics-cert\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061580 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-node-log\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061595 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-multus-cni-dir\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061578 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94e33bbc-9879-4500-abd0-a3cde0baf87f-hosts-file\") pod \"node-resolver-p9fv9\" (UID: \"94e33bbc-9879-4500-abd0-a3cde0baf87f\") " pod="openshift-dns/node-resolver-p9fv9" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061640 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-run\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061695 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-modprobe-d\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061704 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-os-release\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061741 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-etc-kubernetes\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061749 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-multus-socket-dir-parent\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.062659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061756 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-run-netns\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dce67161-8f6a-4e0d-86ab-62245da6f838-os-release\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:58.061816 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061823 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-sys-fs\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:58.061859 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs podName:e460fb8e-afe6-4628-b08e-514c648282c3 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:58.561829249 +0000 UTC m=+3.059336681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs") pod "network-metrics-daemon-2wqk7" (UID: "e460fb8e-afe6-4628-b08e-514c648282c3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061856 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dce67161-8f6a-4e0d-86ab-62245da6f838-cni-binary-copy\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061865 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-sys-fs\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061892 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dce67161-8f6a-4e0d-86ab-62245da6f838-os-release\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:58.061897 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret podName:21a0247b-c2c6-4f3e-a619-2eaaf17a2a06 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:58.561882002 +0000 UTC m=+3.059389430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret") pod "global-pull-secret-syncer-5zw6v" (UID: "21a0247b-c2c6-4f3e-a619-2eaaf17a2a06") : object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061951 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-hostroot\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061954 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dce67161-8f6a-4e0d-86ab-62245da6f838-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061953 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-lib-modules\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.061989 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-run-systemd\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.062007 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-tuned\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.062038 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-lib-modules\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.062038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-log-socket\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.063544 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.062074 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.062098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dce67161-8f6a-4e0d-86ab-62245da6f838-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.062106 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-run-ovn-kubernetes\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.062114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e33bbc-9879-4500-abd0-a3cde0baf87f-tmp-dir\") pod \"node-resolver-p9fv9\" (UID: \"94e33bbc-9879-4500-abd0-a3cde0baf87f\") " pod="openshift-dns/node-resolver-p9fv9" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.062077 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-log-socket\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.062139 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.062179 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbsn\" (UniqueName: \"kubernetes.io/projected/0433a177-3a7d-48d8-97b6-f48375cceb33-kube-api-access-rhbsn\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.062196 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.062211 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90be7f2f-8c43-40b8-9126-11e4f0f72f96-ovnkube-script-lib\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.062267 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dce67161-8f6a-4e0d-86ab-62245da6f838-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.062922 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e33bbc-9879-4500-abd0-a3cde0baf87f-tmp-dir\") pod \"node-resolver-p9fv9\" (UID: \"94e33bbc-9879-4500-abd0-a3cde0baf87f\") " pod="openshift-dns/node-resolver-p9fv9" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063101 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063134 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/90be7f2f-8c43-40b8-9126-11e4f0f72f96-ovnkube-script-lib\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8xr\" (UniqueName: \"kubernetes.io/projected/90be7f2f-8c43-40b8-9126-11e4f0f72f96-kube-api-access-nr8xr\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063242 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdmm7\" (UniqueName: \"kubernetes.io/projected/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-kube-api-access-fdmm7\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063392 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-kubelet\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.064371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dce67161-8f6a-4e0d-86ab-62245da6f838-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5rs2\" (UniqueName: \"kubernetes.io/projected/dce67161-8f6a-4e0d-86ab-62245da6f838-kube-api-access-x5rs2\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063487 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-device-dir\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-systemd\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-var-lib-cni-bin\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063580 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-systemd-units\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063610 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-slash\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90be7f2f-8c43-40b8-9126-11e4f0f72f96-ovnkube-config\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063666 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-kubernetes\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-sysctl-conf\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063744 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/21b82000-0ecd-486b-9e01-b98cf640590b-device-dir\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063799 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-kubelet\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063800 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-var-lib-cni-multus\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063859 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-var-lib-openvswitch\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-cni-netd\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063918 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gplq\" (UniqueName: \"kubernetes.io/projected/94e33bbc-9879-4500-abd0-a3cde0baf87f-kube-api-access-2gplq\") pod \"node-resolver-p9fv9\" (UID: \"94e33bbc-9879-4500-abd0-a3cde0baf87f\") " pod="openshift-dns/node-resolver-p9fv9" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063960 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-host\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.065230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-var-lib-cni-multus\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.063992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-run-multus-certs\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.064040 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-systemd\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.064025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-run-k8s-cni-cncf-io\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.064090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-var-lib-kubelet\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.064117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-run-openvswitch\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.064126 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-var-lib-cni-bin\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.064147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-run-ovn\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.064219 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-systemd-units\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.064265 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-slash\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.064285 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-sys\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.064316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-etc-openvswitch\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.064348 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-dbus\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.064524 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-dbus\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.064774 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/90be7f2f-8c43-40b8-9126-11e4f0f72f96-ovnkube-config\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.064853 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-kubernetes\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.064987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-var-lib-openvswitch\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.065038 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dce67161-8f6a-4e0d-86ab-62245da6f838-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.066045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.065077 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-sysctl-conf\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.066908 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.065077 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-host-cni-netd\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.066908 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.065138 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-run-openvswitch\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.066908 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.065177 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-sys\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.066908 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.065192 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-run-ovn\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.066908 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.065227 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-tmp\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.066908 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.065274 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/90be7f2f-8c43-40b8-9126-11e4f0f72f96-etc-openvswitch\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.066908 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.065307 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-run-multus-certs\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.066908 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.065323 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-run-k8s-cni-cncf-io\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.066908 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.065366 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-host\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.066908 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.065372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0433a177-3a7d-48d8-97b6-f48375cceb33-host-var-lib-kubelet\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.067384 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.067294 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-etc-tuned\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.067442 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.067421 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/90be7f2f-8c43-40b8-9126-11e4f0f72f96-ovn-node-metrics-cert\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.071447 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.071421 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx92l\" (UniqueName: \"kubernetes.io/projected/21b82000-0ecd-486b-9e01-b98cf640590b-kube-api-access-kx92l\") pod \"aws-ebs-csi-driver-node-4lh9k\" (UID: \"21b82000-0ecd-486b-9e01-b98cf640590b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:58.072545 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.072520 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbsn\" (UniqueName: \"kubernetes.io/projected/0433a177-3a7d-48d8-97b6-f48375cceb33-kube-api-access-rhbsn\") pod \"multus-llddr\" (UID: \"0433a177-3a7d-48d8-97b6-f48375cceb33\") " pod="openshift-multus/multus-llddr" Jul 22 09:17:58.072768 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.072742 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrq8n\" (UniqueName: \"kubernetes.io/projected/e460fb8e-afe6-4628-b08e-514c648282c3-kube-api-access-zrq8n\") pod \"network-metrics-daemon-2wqk7\" (UID: \"e460fb8e-afe6-4628-b08e-514c648282c3\") " pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:17:58.072852 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.072786 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gplq\" (UniqueName: \"kubernetes.io/projected/94e33bbc-9879-4500-abd0-a3cde0baf87f-kube-api-access-2gplq\") pod \"node-resolver-p9fv9\" (UID: \"94e33bbc-9879-4500-abd0-a3cde0baf87f\") " pod="openshift-dns/node-resolver-p9fv9" Jul 22 09:17:58.073090 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.073072 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdmm7\" (UniqueName: \"kubernetes.io/projected/3f8b5a27-02e1-4222-bb96-1f8439f4aed5-kube-api-access-fdmm7\") pod \"tuned-n48m7\" (UID: \"3f8b5a27-02e1-4222-bb96-1f8439f4aed5\") " pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.073543 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.073526 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8xr\" (UniqueName: \"kubernetes.io/projected/90be7f2f-8c43-40b8-9126-11e4f0f72f96-kube-api-access-nr8xr\") pod \"ovnkube-node-mdr8h\" (UID: \"90be7f2f-8c43-40b8-9126-11e4f0f72f96\") " pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.074076 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.074060 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5rs2\" (UniqueName: \"kubernetes.io/projected/dce67161-8f6a-4e0d-86ab-62245da6f838-kube-api-access-x5rs2\") pod \"multus-additional-cni-plugins-x8j5l\" (UID: \"dce67161-8f6a-4e0d-86ab-62245da6f838\") " pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.158465 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.158404 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-phzcz" Jul 22 09:17:58.165009 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.164995 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-98sq4" Jul 22 09:17:58.166673 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:58.166572 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc007a031_f135_4b00_8075_b9e362cb09f5.slice/crio-64de6bca3fa8fae954f76138519b61d57ab81ed6534444a0e382375fc04a76b8 WatchSource:0}: Error finding container 64de6bca3fa8fae954f76138519b61d57ab81ed6534444a0e382375fc04a76b8: Status 404 returned error can't find the container with id 64de6bca3fa8fae954f76138519b61d57ab81ed6534444a0e382375fc04a76b8 Jul 22 09:17:58.171706 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:58.171686 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4fc604_5bcb_4274_a20d_5710d075dce3.slice/crio-8bf90d8472f386f65e06e27cefbceb14d1a802abbfa63adb9887fc145489591f WatchSource:0}: Error finding container 8bf90d8472f386f65e06e27cefbceb14d1a802abbfa63adb9887fc145489591f: Status 404 returned error can't find the container with id 8bf90d8472f386f65e06e27cefbceb14d1a802abbfa63adb9887fc145489591f Jul 22 09:17:58.174503 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.174488 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xqb5q" Jul 22 09:17:58.177063 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.177046 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p9fv9" Jul 22 09:17:58.181769 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.181738 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-llddr" Jul 22 09:17:58.182055 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:58.182037 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9425446d_1c7a_419d_8020_59828616976a.slice/crio-8464f226dfe4717aa338054baec5eac9e0d46bdbe7f1c2060f835e9271427ced WatchSource:0}: Error finding container 8464f226dfe4717aa338054baec5eac9e0d46bdbe7f1c2060f835e9271427ced: Status 404 returned error can't find the container with id 8464f226dfe4717aa338054baec5eac9e0d46bdbe7f1c2060f835e9271427ced Jul 22 09:17:58.184989 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:58.184923 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e33bbc_9879_4500_abd0_a3cde0baf87f.slice/crio-67f127233acd2bcb487f6ef3a9badee0015ed8a2a16c1bfdbddbc24dc4a9fcc9 WatchSource:0}: Error finding container 67f127233acd2bcb487f6ef3a9badee0015ed8a2a16c1bfdbddbc24dc4a9fcc9: Status 404 returned error can't find the container with id 67f127233acd2bcb487f6ef3a9badee0015ed8a2a16c1bfdbddbc24dc4a9fcc9 Jul 22 09:17:58.186875 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.186861 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:17:58.191074 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:58.191053 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0433a177_3a7d_48d8_97b6_f48375cceb33.slice/crio-1c083b62985d7a327ebbb89016684cf231eb5d5fb2087b2e8f3bbed6f4d8f666 WatchSource:0}: Error finding container 1c083b62985d7a327ebbb89016684cf231eb5d5fb2087b2e8f3bbed6f4d8f666: Status 404 returned error can't find the container with id 1c083b62985d7a327ebbb89016684cf231eb5d5fb2087b2e8f3bbed6f4d8f666 Jul 22 09:17:58.191704 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.191687 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x8j5l" Jul 22 09:17:58.197131 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.197112 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" Jul 22 09:17:58.197370 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:58.197352 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90be7f2f_8c43_40b8_9126_11e4f0f72f96.slice/crio-94433627000cd6f10072f64849b20e99757036b8eaaf134a522856882903406e WatchSource:0}: Error finding container 94433627000cd6f10072f64849b20e99757036b8eaaf134a522856882903406e: Status 404 returned error can't find the container with id 94433627000cd6f10072f64849b20e99757036b8eaaf134a522856882903406e Jul 22 09:17:58.200141 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:58.200123 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce67161_8f6a_4e0d_86ab_62245da6f838.slice/crio-e6e3d096b4bbda2def291c3970c9ecece5459afc9e52f9fe8d4869a1f1c254e6 WatchSource:0}: Error finding container e6e3d096b4bbda2def291c3970c9ecece5459afc9e52f9fe8d4869a1f1c254e6: Status 404 returned error can't find the container with id e6e3d096b4bbda2def291c3970c9ecece5459afc9e52f9fe8d4869a1f1c254e6 Jul 22 09:17:58.201105 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.201077 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n48m7" Jul 22 09:17:58.204379 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:58.204356 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b82000_0ecd_486b_9e01_b98cf640590b.slice/crio-9610a04f277b205980ebb652af11b2208c4a85b3250665cf2c2f78b4c5ca8169 WatchSource:0}: Error finding container 9610a04f277b205980ebb652af11b2208c4a85b3250665cf2c2f78b4c5ca8169: Status 404 returned error can't find the container with id 9610a04f277b205980ebb652af11b2208c4a85b3250665cf2c2f78b4c5ca8169 Jul 22 09:17:58.208173 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:58.208154 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8b5a27_02e1_4222_bb96_1f8439f4aed5.slice/crio-bff579a4dd7dede000034ca65ee92d48cd363d8781faed56ef0b2a4a76a45e20 WatchSource:0}: Error finding container bff579a4dd7dede000034ca65ee92d48cd363d8781faed56ef0b2a4a76a45e20: Status 404 returned error can't find the container with id bff579a4dd7dede000034ca65ee92d48cd363d8781faed56ef0b2a4a76a45e20 Jul 22 09:17:58.466788 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.466762 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74lfl\" (UniqueName: \"kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl\") pod \"network-check-target-km7cq\" (UID: \"ec80e86d-c05c-439a-ad34-c44e4009f060\") " pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:17:58.466969 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:58.466917 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 09:17:58.466969 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:58.466952 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 09:17:58.466969 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:58.466965 2564 projected.go:196] Error preparing data for projected volume kube-api-access-74lfl for pod openshift-network-diagnostics/network-check-target-km7cq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:17:58.467130 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:58.467024 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl podName:ec80e86d-c05c-439a-ad34-c44e4009f060 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:59.467006058 +0000 UTC m=+3.964513488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-74lfl" (UniqueName: "kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl") pod "network-check-target-km7cq" (UID: "ec80e86d-c05c-439a-ad34-c44e4009f060") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:17:58.567576 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.567536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:17:58.567704 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.567619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs\") pod \"network-metrics-daemon-2wqk7\" (UID: \"e460fb8e-afe6-4628-b08e-514c648282c3\") " pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:17:58.567765 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:58.567738 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:58.567821 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:58.567793 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs podName:e460fb8e-afe6-4628-b08e-514c648282c3 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:59.567776849 +0000 UTC m=+4.065284266 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs") pod "network-metrics-daemon-2wqk7" (UID: "e460fb8e-afe6-4628-b08e-514c648282c3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:58.567882 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:58.567827 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:58.567929 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:58.567885 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret podName:21a0247b-c2c6-4f3e-a619-2eaaf17a2a06 nodeName:}" failed. No retries permitted until 2026-07-22 09:17:59.567869077 +0000 UTC m=+4.065376508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret") pod "global-pull-secret-syncer-5zw6v" (UID: "21a0247b-c2c6-4f3e-a619-2eaaf17a2a06") : object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:58.571537 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:17:58.571511 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0fc6bf224341812e0ad602c5f4c985.slice/crio-6cc06e5004f3fba9d90addd60a535595456426d46a1adf8e5cfe24ef65dc07d3 WatchSource:0}: Error finding container 6cc06e5004f3fba9d90addd60a535595456426d46a1adf8e5cfe24ef65dc07d3: Status 404 returned error can't find the container with id 6cc06e5004f3fba9d90addd60a535595456426d46a1adf8e5cfe24ef65dc07d3 Jul 22 09:17:58.906846 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.906757 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 09:12:57 +0000 UTC" deadline="2028-05-08 19:40:12.410349772 +0000 UTC" Jul 22 09:17:58.906846 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:58.906795 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15754h22m13.503559533s" Jul 22 09:17:59.008629 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.008596 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:17:59.008803 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:59.008733 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:17:59.017403 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.017362 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xqb5q" event={"ID":"9425446d-1c7a-419d-8020-59828616976a","Type":"ContainerStarted","Data":"8464f226dfe4717aa338054baec5eac9e0d46bdbe7f1c2060f835e9271427ced"} Jul 22 09:17:59.019512 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.019383 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-98sq4" event={"ID":"9d4fc604-5bcb-4274-a20d-5710d075dce3","Type":"ContainerStarted","Data":"8bf90d8472f386f65e06e27cefbceb14d1a802abbfa63adb9887fc145489591f"} Jul 22 09:17:59.022047 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.022015 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-phzcz" event={"ID":"c007a031-f135-4b00-8075-b9e362cb09f5","Type":"ContainerStarted","Data":"64de6bca3fa8fae954f76138519b61d57ab81ed6534444a0e382375fc04a76b8"} Jul 22 09:17:59.023398 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.023373 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal" event={"ID":"7e0fc6bf224341812e0ad602c5f4c985","Type":"ContainerStarted","Data":"6cc06e5004f3fba9d90addd60a535595456426d46a1adf8e5cfe24ef65dc07d3"} Jul 22 09:17:59.024647 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.024623 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-llddr" event={"ID":"0433a177-3a7d-48d8-97b6-f48375cceb33","Type":"ContainerStarted","Data":"1c083b62985d7a327ebbb89016684cf231eb5d5fb2087b2e8f3bbed6f4d8f666"} Jul 22 09:17:59.026045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.026022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p9fv9" event={"ID":"94e33bbc-9879-4500-abd0-a3cde0baf87f","Type":"ContainerStarted","Data":"67f127233acd2bcb487f6ef3a9badee0015ed8a2a16c1bfdbddbc24dc4a9fcc9"} Jul 22 09:17:59.027703 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.027679 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n48m7" event={"ID":"3f8b5a27-02e1-4222-bb96-1f8439f4aed5","Type":"ContainerStarted","Data":"bff579a4dd7dede000034ca65ee92d48cd363d8781faed56ef0b2a4a76a45e20"} Jul 22 09:17:59.029722 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.029641 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" event={"ID":"21b82000-0ecd-486b-9e01-b98cf640590b","Type":"ContainerStarted","Data":"9610a04f277b205980ebb652af11b2208c4a85b3250665cf2c2f78b4c5ca8169"} Jul 22 09:17:59.031176 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.031131 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8j5l" event={"ID":"dce67161-8f6a-4e0d-86ab-62245da6f838","Type":"ContainerStarted","Data":"e6e3d096b4bbda2def291c3970c9ecece5459afc9e52f9fe8d4869a1f1c254e6"} Jul 22 09:17:59.032886 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.032844 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" event={"ID":"90be7f2f-8c43-40b8-9126-11e4f0f72f96","Type":"ContainerStarted","Data":"94433627000cd6f10072f64849b20e99757036b8eaaf134a522856882903406e"} Jul 22 09:17:59.473433 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.473399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74lfl\" (UniqueName: \"kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl\") pod \"network-check-target-km7cq\" (UID: \"ec80e86d-c05c-439a-ad34-c44e4009f060\") " pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:17:59.473647 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:59.473628 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 09:17:59.473720 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:59.473653 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 09:17:59.473720 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:59.473666 2564 projected.go:196] Error preparing data for projected volume kube-api-access-74lfl for pod openshift-network-diagnostics/network-check-target-km7cq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:17:59.473839 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:59.473723 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl podName:ec80e86d-c05c-439a-ad34-c44e4009f060 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:01.473705205 +0000 UTC m=+5.971212635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-74lfl" (UniqueName: "kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl") pod "network-check-target-km7cq" (UID: "ec80e86d-c05c-439a-ad34-c44e4009f060") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:17:59.574677 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.574629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs\") pod \"network-metrics-daemon-2wqk7\" (UID: \"e460fb8e-afe6-4628-b08e-514c648282c3\") " pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:17:59.574813 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.574710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:17:59.574876 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:59.574838 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:59.574930 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:59.574900 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret podName:21a0247b-c2c6-4f3e-a619-2eaaf17a2a06 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:01.574882249 +0000 UTC m=+6.072389668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret") pod "global-pull-secret-syncer-5zw6v" (UID: "21a0247b-c2c6-4f3e-a619-2eaaf17a2a06") : object "kube-system"/"original-pull-secret" not registered Jul 22 09:17:59.575398 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:59.575372 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:59.575486 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:17:59.575427 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs podName:e460fb8e-afe6-4628-b08e-514c648282c3 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:01.575412241 +0000 UTC m=+6.072919660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs") pod "network-metrics-daemon-2wqk7" (UID: "e460fb8e-afe6-4628-b08e-514c648282c3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:17:59.868787 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.868701 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 09:17:59.907017 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.906974 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 09:12:57 +0000 UTC" deadline="2028-03-14 03:42:32.718706462 +0000 UTC" Jul 22 09:17:59.907017 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:17:59.907013 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14418h24m32.811698083s" Jul 22 09:18:00.009611 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:00.009578 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:00.009791 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:00.009577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:00.009860 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:00.009771 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:00.010368 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:00.010338 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:01.008370 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:01.008332 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:01.008855 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:01.008472 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:01.489127 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:01.488598 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74lfl\" (UniqueName: \"kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl\") pod \"network-check-target-km7cq\" (UID: \"ec80e86d-c05c-439a-ad34-c44e4009f060\") " pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:01.489127 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:01.489010 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 09:18:01.489127 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:01.489031 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 09:18:01.489127 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:01.489045 2564 projected.go:196] Error preparing data for projected volume kube-api-access-74lfl for pod openshift-network-diagnostics/network-check-target-km7cq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:18:01.489127 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:01.489096 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl podName:ec80e86d-c05c-439a-ad34-c44e4009f060 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:05.489077685 +0000 UTC m=+9.986585103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-74lfl" (UniqueName: "kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl") pod "network-check-target-km7cq" (UID: "ec80e86d-c05c-439a-ad34-c44e4009f060") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:18:01.591262 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:01.590210 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:01.591262 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:01.590389 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs\") pod \"network-metrics-daemon-2wqk7\" (UID: \"e460fb8e-afe6-4628-b08e-514c648282c3\") " pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:01.591262 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:01.590551 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:18:01.591262 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:01.590615 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs podName:e460fb8e-afe6-4628-b08e-514c648282c3 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:05.590595736 +0000 UTC m=+10.088103153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs") pod "network-metrics-daemon-2wqk7" (UID: "e460fb8e-afe6-4628-b08e-514c648282c3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:18:01.591262 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:01.591027 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 09:18:01.591262 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:01.591074 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret podName:21a0247b-c2c6-4f3e-a619-2eaaf17a2a06 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:05.5910592 +0000 UTC m=+10.088566618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret") pod "global-pull-secret-syncer-5zw6v" (UID: "21a0247b-c2c6-4f3e-a619-2eaaf17a2a06") : object "kube-system"/"original-pull-secret" not registered Jul 22 09:18:02.009090 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:02.009058 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:02.009622 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:02.009209 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:02.009622 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:02.009257 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:02.009622 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:02.009375 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:02.053792 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:02.052859 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xqb5q" event={"ID":"9425446d-1c7a-419d-8020-59828616976a","Type":"ContainerStarted","Data":"0b4f117a9d56acaba75c49f572f2b87972d136236b265e2d1c1cd84590c23a58"} Jul 22 09:18:02.059953 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:02.059908 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-98sq4" event={"ID":"9d4fc604-5bcb-4274-a20d-5710d075dce3","Type":"ContainerStarted","Data":"2a9c45e39d7fbe8484f5df2d8f086d41766404fa056c7855e87d69138c6782b3"} Jul 22 09:18:02.062388 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:02.062354 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p9fv9" event={"ID":"94e33bbc-9879-4500-abd0-a3cde0baf87f","Type":"ContainerStarted","Data":"b8691577d7e46bfb5a24ea7afd4b84b7a03fa7c913e5ea4fb9612d0644836c1f"} Jul 22 09:18:02.091686 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:02.090285 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xqb5q" podStartSLOduration=3.184871301 podStartE2EDuration="6.090263164s" totalImagesPullingTime="2.905391863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:02.072279726 +0000 UTC m=+6.569787167" watchObservedRunningTime="2026-07-22 09:18:02.090263164 +0000 UTC m=+6.587770601" Jul 22 09:18:02.120243 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:02.119537 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-98sq4" podStartSLOduration=3.202707233 podStartE2EDuration="6.119511428s" totalImagesPullingTime="2.916804195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:02.09149288 +0000 UTC m=+6.589000317" watchObservedRunningTime="2026-07-22 09:18:02.119511428 +0000 UTC m=+6.617018867" Jul 22 09:18:03.008707 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:03.008217 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:03.008707 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:03.008348 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:03.069229 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:03.068484 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-phzcz" event={"ID":"c007a031-f135-4b00-8075-b9e362cb09f5","Type":"ContainerStarted","Data":"f7a717e987640ac06e77ffafe8d50d1a4af291cb88296ccb9cda3ab8097399ab"} Jul 22 09:18:03.088821 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:03.088762 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-p9fv9" podStartSLOduration=4.185244999 podStartE2EDuration="7.088740592s" totalImagesPullingTime="2.903495593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:02.120218522 +0000 UTC m=+6.617725959" watchObservedRunningTime="2026-07-22 09:18:03.088740592 +0000 UTC m=+7.586248034" Jul 22 09:18:04.010810 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:04.010241 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:04.010810 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:04.010420 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:04.010810 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:04.010445 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:04.010810 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:04.010559 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:05.008266 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:05.008233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:05.008780 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:05.008374 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:05.521836 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:05.521764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74lfl\" (UniqueName: \"kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl\") pod \"network-check-target-km7cq\" (UID: \"ec80e86d-c05c-439a-ad34-c44e4009f060\") " pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:05.522025 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:05.521971 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 09:18:05.522025 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:05.521996 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 09:18:05.522025 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:05.522009 2564 projected.go:196] Error preparing data for projected volume kube-api-access-74lfl for pod openshift-network-diagnostics/network-check-target-km7cq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:18:05.522166 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:05.522072 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl podName:ec80e86d-c05c-439a-ad34-c44e4009f060 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:13.522051652 +0000 UTC m=+18.019559068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-74lfl" (UniqueName: "kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl") pod "network-check-target-km7cq" (UID: "ec80e86d-c05c-439a-ad34-c44e4009f060") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:18:05.527764 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:05.527742 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xqb5q" Jul 22 09:18:05.528395 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:05.528375 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xqb5q" Jul 22 09:18:05.547146 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:05.546982 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-phzcz" podStartSLOduration=7.625832874 podStartE2EDuration="10.546963566s" totalImagesPullingTime="2.921130692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:03.084479776 +0000 UTC m=+7.581987205" watchObservedRunningTime="2026-07-22 09:18:05.546963566 +0000 UTC m=+10.044470996" Jul 22 09:18:05.623085 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:05.623050 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs\") pod \"network-metrics-daemon-2wqk7\" (UID: \"e460fb8e-afe6-4628-b08e-514c648282c3\") " pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:05.623246 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:05.623132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:05.623246 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:05.623199 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:18:05.623380 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:05.623265 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs podName:e460fb8e-afe6-4628-b08e-514c648282c3 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:13.623246964 +0000 UTC m=+18.120754398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs") pod "network-metrics-daemon-2wqk7" (UID: "e460fb8e-afe6-4628-b08e-514c648282c3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:18:05.623657 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:05.623633 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 09:18:05.623740 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:05.623701 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret podName:21a0247b-c2c6-4f3e-a619-2eaaf17a2a06 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:13.623683441 +0000 UTC m=+18.121190862 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret") pod "global-pull-secret-syncer-5zw6v" (UID: "21a0247b-c2c6-4f3e-a619-2eaaf17a2a06") : object "kube-system"/"original-pull-secret" not registered Jul 22 09:18:06.009877 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:06.009368 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:06.009877 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:06.009514 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:06.009877 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:06.009555 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:06.009877 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:06.009679 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:07.008865 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:07.008820 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:07.009078 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:07.008984 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:08.008824 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:08.008786 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:08.009234 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:08.008919 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:08.009234 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:08.008980 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:08.009234 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:08.009102 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:09.008982 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:09.008945 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:09.009441 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:09.009089 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:10.008573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:10.008541 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:10.008573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:10.008564 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:10.008803 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:10.008668 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:10.008859 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:10.008794 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:11.009022 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:11.008994 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:11.009418 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:11.009108 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:12.008647 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:12.008600 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:12.008647 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:12.008647 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:12.008886 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:12.008776 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:12.008961 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:12.008908 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:13.008880 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:13.008846 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:13.009334 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:13.009065 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:13.582078 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:13.582034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74lfl\" (UniqueName: \"kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl\") pod \"network-check-target-km7cq\" (UID: \"ec80e86d-c05c-439a-ad34-c44e4009f060\") " pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:13.582254 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:13.582208 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 09:18:13.582254 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:13.582229 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 09:18:13.582254 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:13.582239 2564 projected.go:196] Error preparing data for projected volume kube-api-access-74lfl for pod openshift-network-diagnostics/network-check-target-km7cq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:18:13.582391 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:13.582298 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl podName:ec80e86d-c05c-439a-ad34-c44e4009f060 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:29.582279074 +0000 UTC m=+34.079786504 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-74lfl" (UniqueName: "kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl") pod "network-check-target-km7cq" (UID: "ec80e86d-c05c-439a-ad34-c44e4009f060") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:18:13.683172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:13.683129 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs\") pod \"network-metrics-daemon-2wqk7\" (UID: \"e460fb8e-afe6-4628-b08e-514c648282c3\") " pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:13.683371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:13.683207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:13.683371 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:13.683343 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 09:18:13.683486 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:13.683399 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret podName:21a0247b-c2c6-4f3e-a619-2eaaf17a2a06 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:29.683384721 +0000 UTC m=+34.180892138 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret") pod "global-pull-secret-syncer-5zw6v" (UID: "21a0247b-c2c6-4f3e-a619-2eaaf17a2a06") : object "kube-system"/"original-pull-secret" not registered Jul 22 09:18:13.683486 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:13.683344 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:18:13.683599 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:13.683492 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs podName:e460fb8e-afe6-4628-b08e-514c648282c3 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:29.683470653 +0000 UTC m=+34.180978073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs") pod "network-metrics-daemon-2wqk7" (UID: "e460fb8e-afe6-4628-b08e-514c648282c3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:18:14.009260 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:14.009226 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:14.009718 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:14.009369 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:14.009718 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:14.009422 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:14.009718 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:14.009557 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:15.008455 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:15.008424 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:15.008612 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:15.008535 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:15.872022 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:15.871984 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xqb5q" Jul 22 09:18:15.872481 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:15.872143 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 22 09:18:15.873076 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:15.873039 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xqb5q" Jul 22 09:18:16.009157 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:16.009120 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:16.009340 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:16.009242 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:16.009340 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:16.009244 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:16.009442 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:16.009371 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:17.009132 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:17.009104 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:17.009568 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:17.009223 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:18.008717 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.008445 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:18.008849 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.008509 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:18.008849 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:18.008825 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:18.008971 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:18.008898 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:18.098803 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.098655 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-55.ec2.internal" event={"ID":"fceedcd76d463c4089e6465fad28b1d6","Type":"ContainerStarted","Data":"008bad33b4ddf8338199e537fa8d14bf440d75e8f7ccf4fe92adbc58d09003cf"} Jul 22 09:18:18.100094 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.100075 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n48m7" event={"ID":"3f8b5a27-02e1-4222-bb96-1f8439f4aed5","Type":"ContainerStarted","Data":"2b529b7969a06b0077f7e20fec5a08e2ce913f9bd259c056fde815bc6f574dd1"} Jul 22 09:18:18.101406 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.101386 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" event={"ID":"21b82000-0ecd-486b-9e01-b98cf640590b","Type":"ContainerStarted","Data":"f4728daa23bc9c86b89ec631c9f44525eaac206ff41177154d0acd5b2d9803c1"} Jul 22 09:18:18.102821 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.102796 2564 generic.go:356] "Generic (PLEG): container finished" podID="dce67161-8f6a-4e0d-86ab-62245da6f838" containerID="e5e01f03c44c58b59d9fed0a90640b9085f48b90589b5f52038a71f3120fba50" exitCode=0 Jul 22 09:18:18.102919 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.102878 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8j5l" event={"ID":"dce67161-8f6a-4e0d-86ab-62245da6f838","Type":"ContainerDied","Data":"e5e01f03c44c58b59d9fed0a90640b9085f48b90589b5f52038a71f3120fba50"} Jul 22 09:18:18.105228 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.105209 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:18:18.105524 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.105503 2564 generic.go:356] "Generic (PLEG): container finished" podID="90be7f2f-8c43-40b8-9126-11e4f0f72f96" containerID="cc6bcca7419dc7a4b72518035074c1bff2a867392b5439f28f5b772098d7a9e3" exitCode=1 Jul 22 09:18:18.105577 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.105566 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" event={"ID":"90be7f2f-8c43-40b8-9126-11e4f0f72f96","Type":"ContainerStarted","Data":"83fe5f90919ef6cae44fd5aee276ac7f460bd75ae4d6fe6a32ca96af2d699ade"} Jul 22 09:18:18.105643 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.105624 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" event={"ID":"90be7f2f-8c43-40b8-9126-11e4f0f72f96","Type":"ContainerStarted","Data":"d2271aa441a9e62d344baa7f47881b90442f86347cd1c0be5490533835900128"} Jul 22 09:18:18.105730 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.105675 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" event={"ID":"90be7f2f-8c43-40b8-9126-11e4f0f72f96","Type":"ContainerDied","Data":"cc6bcca7419dc7a4b72518035074c1bff2a867392b5439f28f5b772098d7a9e3"} Jul 22 09:18:18.105730 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.105707 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" event={"ID":"90be7f2f-8c43-40b8-9126-11e4f0f72f96","Type":"ContainerStarted","Data":"78de0b81ddb38adb4e996700e307ec5a2034018ccca1a76cb5a7c6a94be33143"} Jul 22 09:18:18.106860 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.106842 2564 generic.go:356] "Generic (PLEG): container finished" podID="7e0fc6bf224341812e0ad602c5f4c985" containerID="cf99da061b020a6936584d7676f0b7d0a7cc1956ac0bace4824ed0d7f8f448ff" exitCode=0 Jul 22 09:18:18.106918 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.106895 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal" event={"ID":"7e0fc6bf224341812e0ad602c5f4c985","Type":"ContainerDied","Data":"cf99da061b020a6936584d7676f0b7d0a7cc1956ac0bace4824ed0d7f8f448ff"} Jul 22 09:18:18.108348 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.108275 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-llddr" event={"ID":"0433a177-3a7d-48d8-97b6-f48375cceb33","Type":"ContainerStarted","Data":"bfcb99444e5f219a076ad09f94d9f40e921e381c99e7b86421e37edac204db69"} Jul 22 09:18:18.150302 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.150250 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-55.ec2.internal" podStartSLOduration=21.150230104 podStartE2EDuration="21.150230104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:18.123653726 +0000 UTC m=+22.621161163" watchObservedRunningTime="2026-07-22 09:18:18.150230104 +0000 UTC m=+22.647737541" Jul 22 09:18:18.209476 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:18.209423 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-llddr" podStartSLOduration=2.7636622920000002 podStartE2EDuration="22.209403611s" totalImagesPullingTime="19.445741319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:18.20840624 +0000 UTC m=+22.705913677" watchObservedRunningTime="2026-07-22 09:18:18.209403611 +0000 UTC m=+22.706911047" Jul 22 09:18:19.008892 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:19.008846 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:19.009074 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:19.009002 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:19.113853 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:19.113828 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:18:19.114346 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:19.114322 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" event={"ID":"90be7f2f-8c43-40b8-9126-11e4f0f72f96","Type":"ContainerStarted","Data":"36446db0305849ada0f65a42e26c27eef66c5116bd7e633508a50549b3356830"} Jul 22 09:18:19.114419 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:19.114405 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" event={"ID":"90be7f2f-8c43-40b8-9126-11e4f0f72f96","Type":"ContainerStarted","Data":"4aaed939d93b696449aa01acc0dee55bb7e14c0a6f2a7e77280cd54896f2cc18"} Jul 22 09:18:19.116140 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:19.116115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal" event={"ID":"7e0fc6bf224341812e0ad602c5f4c985","Type":"ContainerStarted","Data":"64d90b3fe787cfe849b76e2bf6695fc7fea09d545d750a03547f62e0ff14b13a"} Jul 22 09:18:19.132406 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:19.132258 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n48m7" podStartSLOduration=3.743552957 podStartE2EDuration="23.132240254s" totalImagesPullingTime="19.388687297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:18.263839134 +0000 UTC m=+22.761346571" watchObservedRunningTime="2026-07-22 09:18:19.132240254 +0000 UTC m=+23.629747694" Jul 22 09:18:19.132666 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:19.132634 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-55.ec2.internal" podStartSLOduration=22.132625943 podStartE2EDuration="22.132625943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:19.131905949 +0000 UTC m=+23.629413386" watchObservedRunningTime="2026-07-22 09:18:19.132625943 +0000 UTC m=+23.630133380" Jul 22 09:18:19.147961 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:19.147756 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 09:18:19.922849 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:19.922737 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T09:18:19.147956588Z","UUID":"bfd309dc-118b-421e-9369-56926ab9c765","Handler":null,"Name":"","Endpoint":""} Jul 22 09:18:19.924922 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:19.924899 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 09:18:19.925064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:19.924964 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 09:18:20.008494 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:20.008449 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:20.008665 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:20.008604 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:20.008665 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:20.008620 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:20.008783 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:20.008723 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:20.121056 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:20.121020 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" event={"ID":"21b82000-0ecd-486b-9e01-b98cf640590b","Type":"ContainerStarted","Data":"d25a3541b2479e9e6ae47f4af745f52b6fd1c4b490d8a0e001c3006963903441"} Jul 22 09:18:20.121453 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:20.121082 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" event={"ID":"21b82000-0ecd-486b-9e01-b98cf640590b","Type":"ContainerStarted","Data":"8e16da85c8e2174d1aa4cbfdda9cf60ebd283e28e894d68f3df3ac10e327beb1"} Jul 22 09:18:20.140149 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:20.140107 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4lh9k" podStartSLOduration=2.666392824 podStartE2EDuration="24.140087773s" totalImagesPullingTime="21.473694949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:20.139753809 +0000 UTC m=+24.637261245" watchObservedRunningTime="2026-07-22 09:18:20.140087773 +0000 UTC m=+24.637595210" Jul 22 09:18:21.009085 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:21.009052 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:21.009263 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:21.009198 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:21.126843 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:21.126814 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:18:21.127289 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:21.127262 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" event={"ID":"90be7f2f-8c43-40b8-9126-11e4f0f72f96","Type":"ContainerStarted","Data":"9f9ea834c6a364e3da6dad926cc0969c293153b190fbf7526ed1eedc9c233f21"} Jul 22 09:18:22.009274 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:22.009238 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:22.009460 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:22.009391 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:22.009460 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:22.009444 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:22.009584 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:22.009556 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:23.009008 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:23.008977 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:23.009682 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:23.009078 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:23.133226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:23.133194 2564 generic.go:356] "Generic (PLEG): container finished" podID="dce67161-8f6a-4e0d-86ab-62245da6f838" containerID="cb5de1c8020938611e3c286f673e500f0eafd8dde91bdd1aa0320fadea746386" exitCode=0 Jul 22 09:18:23.133372 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:23.133269 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8j5l" event={"ID":"dce67161-8f6a-4e0d-86ab-62245da6f838","Type":"ContainerDied","Data":"cb5de1c8020938611e3c286f673e500f0eafd8dde91bdd1aa0320fadea746386"} Jul 22 09:18:23.136882 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:23.136724 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:18:23.137244 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:23.137222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" event={"ID":"90be7f2f-8c43-40b8-9126-11e4f0f72f96","Type":"ContainerStarted","Data":"70429c66d414f112294101020ec3ba000d15d2075d80fc505cd0a44c8aff8881"} Jul 22 09:18:23.137473 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:23.137456 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:18:23.137570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:23.137513 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:18:23.137631 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:23.137616 2564 scope.go:117] "RemoveContainer" containerID="cc6bcca7419dc7a4b72518035074c1bff2a867392b5439f28f5b772098d7a9e3" Jul 22 09:18:23.153281 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:23.153263 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:18:24.009107 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:24.009076 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:24.009528 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:24.009193 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:24.009528 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:24.009251 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:24.009528 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:24.009395 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:24.141928 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:24.141841 2564 generic.go:356] "Generic (PLEG): container finished" podID="dce67161-8f6a-4e0d-86ab-62245da6f838" containerID="2ae789e25584ecf72d253504df8158e6f585e7f3d66298ab867e110d7b21effc" exitCode=0 Jul 22 09:18:24.142106 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:24.141919 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8j5l" event={"ID":"dce67161-8f6a-4e0d-86ab-62245da6f838","Type":"ContainerDied","Data":"2ae789e25584ecf72d253504df8158e6f585e7f3d66298ab867e110d7b21effc"} Jul 22 09:18:24.145956 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:24.145925 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:18:24.146334 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:24.146305 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" event={"ID":"90be7f2f-8c43-40b8-9126-11e4f0f72f96","Type":"ContainerStarted","Data":"38f1eab28018eb9e50ab9a4122ff6d796bfc6ac0818baed10a1368cde73f0e27"} Jul 22 09:18:24.146584 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:24.146564 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:18:24.160641 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:24.160620 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:18:24.218586 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:24.218540 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" podStartSLOduration=8.742809639 podStartE2EDuration="28.218525817s" totalImagesPullingTime="19.475716178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:24.218118583 +0000 UTC m=+28.715626019" watchObservedRunningTime="2026-07-22 09:18:24.218525817 +0000 UTC m=+28.716033253" Jul 22 09:18:24.648698 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:24.648668 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2wqk7"] Jul 22 09:18:24.648850 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:24.648787 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:24.648895 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:24.648873 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:24.651280 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:24.651253 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-km7cq"] Jul 22 09:18:24.651382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:24.651359 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:24.651483 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:24.651451 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:24.651783 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:24.651765 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5zw6v"] Jul 22 09:18:24.651858 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:24.651846 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:24.651974 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:24.651951 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:26.012239 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:26.010749 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:26.012239 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:26.010998 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:26.195592 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:26.195566 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p9fv9_94e33bbc-9879-4500-abd0-a3cde0baf87f/dns-node-resolver/0.log" Jul 22 09:18:26.970104 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:26.970075 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-98sq4_9d4fc604-5bcb-4274-a20d-5710d075dce3/node-ca/0.log" Jul 22 09:18:27.008308 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:27.008234 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:27.008419 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:27.008384 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:27.008488 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:27.008426 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:27.008634 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:27.008551 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:28.008748 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:28.008570 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:28.009244 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:28.008824 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:28.156649 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:28.156615 2564 generic.go:356] "Generic (PLEG): container finished" podID="dce67161-8f6a-4e0d-86ab-62245da6f838" containerID="d0176c4dec41b26b7a443d0e9c88c3fae6dd95e3ecd8a1d8bcdca68eac6f775a" exitCode=0 Jul 22 09:18:28.156782 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:28.156677 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8j5l" event={"ID":"dce67161-8f6a-4e0d-86ab-62245da6f838","Type":"ContainerDied","Data":"d0176c4dec41b26b7a443d0e9c88c3fae6dd95e3ecd8a1d8bcdca68eac6f775a"} Jul 22 09:18:29.008802 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:29.008768 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:29.009240 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:29.008814 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:29.009240 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:29.008902 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:29.009240 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:29.009039 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:29.596922 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:29.596884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74lfl\" (UniqueName: \"kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl\") pod \"network-check-target-km7cq\" (UID: \"ec80e86d-c05c-439a-ad34-c44e4009f060\") " pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:29.597136 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:29.597063 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 09:18:29.597136 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:29.597083 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 09:18:29.597136 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:29.597094 2564 projected.go:196] Error preparing data for projected volume kube-api-access-74lfl for pod openshift-network-diagnostics/network-check-target-km7cq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:18:29.597266 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:29.597151 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl podName:ec80e86d-c05c-439a-ad34-c44e4009f060 nodeName:}" failed. No retries permitted until 2026-07-22 09:19:01.597133488 +0000 UTC m=+66.094640907 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-74lfl" (UniqueName: "kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl") pod "network-check-target-km7cq" (UID: "ec80e86d-c05c-439a-ad34-c44e4009f060") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 09:18:29.698074 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:29.698038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs\") pod \"network-metrics-daemon-2wqk7\" (UID: \"e460fb8e-afe6-4628-b08e-514c648282c3\") " pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:29.698264 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:29.698094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:29.698264 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:29.698196 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:18:29.698264 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:29.698210 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 09:18:29.698392 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:29.698273 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs podName:e460fb8e-afe6-4628-b08e-514c648282c3 nodeName:}" failed. No retries permitted until 2026-07-22 09:19:01.698251528 +0000 UTC m=+66.195758943 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs") pod "network-metrics-daemon-2wqk7" (UID: "e460fb8e-afe6-4628-b08e-514c648282c3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 09:18:29.698392 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:29.698291 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret podName:21a0247b-c2c6-4f3e-a619-2eaaf17a2a06 nodeName:}" failed. No retries permitted until 2026-07-22 09:19:01.698283206 +0000 UTC m=+66.195790619 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret") pod "global-pull-secret-syncer-5zw6v" (UID: "21a0247b-c2c6-4f3e-a619-2eaaf17a2a06") : object "kube-system"/"original-pull-secret" not registered Jul 22 09:18:30.008320 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:30.008286 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:30.008580 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:30.008426 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:31.008615 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:31.008577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:31.008615 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:31.008596 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:31.009215 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:31.008726 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:31.009215 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:31.008755 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:32.008705 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:32.008673 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:32.009164 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:32.008814 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:33.009106 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:33.009068 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:33.009660 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:33.009172 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:33.009660 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:33.009238 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:33.009660 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:33.009383 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:33.169842 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:33.169812 2564 generic.go:356] "Generic (PLEG): container finished" podID="dce67161-8f6a-4e0d-86ab-62245da6f838" containerID="100b711fcfed3fc9bce2f1fca09c63a367cf510b4de5f6f099b696692bd0505c" exitCode=0 Jul 22 09:18:33.170035 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:33.169868 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8j5l" event={"ID":"dce67161-8f6a-4e0d-86ab-62245da6f838","Type":"ContainerDied","Data":"100b711fcfed3fc9bce2f1fca09c63a367cf510b4de5f6f099b696692bd0505c"} Jul 22 09:18:34.008973 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:34.008920 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:34.009130 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:34.009084 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:34.175228 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:34.175199 2564 generic.go:356] "Generic (PLEG): container finished" podID="dce67161-8f6a-4e0d-86ab-62245da6f838" containerID="c1a425235860036b0333d429806d77947a42ae111b39a973577b78b9327325fb" exitCode=0 Jul 22 09:18:34.175375 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:34.175253 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8j5l" event={"ID":"dce67161-8f6a-4e0d-86ab-62245da6f838","Type":"ContainerDied","Data":"c1a425235860036b0333d429806d77947a42ae111b39a973577b78b9327325fb"} Jul 22 09:18:35.009094 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:35.009061 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:35.009260 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:35.009063 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:35.009260 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:35.009156 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:35.009260 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:35.009226 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:35.180597 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:35.180567 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x8j5l" event={"ID":"dce67161-8f6a-4e0d-86ab-62245da6f838","Type":"ContainerStarted","Data":"03f5bac39b37f03d2f207b8adf32b15da5955613624a8783b8600f60b3a94669"} Jul 22 09:18:35.206047 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:35.206001 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x8j5l" podStartSLOduration=7.410897147 podStartE2EDuration="39.205988728s" totalImagesPullingTime="31.795091581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:35.204650447 +0000 UTC m=+39.702157883" watchObservedRunningTime="2026-07-22 09:18:35.205988728 +0000 UTC m=+39.703496165" Jul 22 09:18:36.009928 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:36.009894 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:36.010316 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:36.010037 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:37.008275 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:37.008236 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:37.008468 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:37.008285 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:37.008468 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:37.008358 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:37.008586 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:37.008464 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:38.009038 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:38.009006 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:38.009404 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:38.009120 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:39.008714 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:39.008675 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:39.008906 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:39.008806 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:39.008906 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:39.008856 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:39.009054 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:39.008971 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:40.008445 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:40.008406 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:40.008614 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:40.008555 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:41.008360 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:41.008320 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:41.008726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:41.008326 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:41.008726 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:41.008433 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:41.008726 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:41.008505 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:42.008509 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:42.008479 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:42.008861 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:42.008587 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:43.008663 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:43.008637 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:43.009022 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:43.008683 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:43.009022 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:43.008743 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:43.009022 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:43.008845 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:44.008704 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:44.008676 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:44.009073 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:44.008776 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:45.008889 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:45.008857 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:45.009292 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:45.008857 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:45.009292 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:45.008974 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5zw6v" podUID="21a0247b-c2c6-4f3e-a619-2eaaf17a2a06" Jul 22 09:18:45.009292 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:45.009031 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-km7cq" podUID="ec80e86d-c05c-439a-ad34-c44e4009f060" Jul 22 09:18:46.009200 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.009166 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:46.009562 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:46.009253 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2wqk7" podUID="e460fb8e-afe6-4628-b08e-514c648282c3" Jul 22 09:18:46.234291 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.234259 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-55.ec2.internal" event="NodeReady" Jul 22 09:18:46.234410 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.234395 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 09:18:46.304395 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.304333 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hntwv"] Jul 22 09:18:46.321071 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.321049 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sgnqf"] Jul 22 09:18:46.321171 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.321160 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.325856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.325838 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 09:18:46.325964 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.325874 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 09:18:46.326483 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.326459 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 09:18:46.327064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.327049 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7wntr\"" Jul 22 09:18:46.327760 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.327741 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 09:18:46.335037 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.335013 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hntwv"] Jul 22 09:18:46.335147 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.335131 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sgnqf" Jul 22 09:18:46.338801 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.338783 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 09:18:46.338877 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.338833 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-v4ns8\"" Jul 22 09:18:46.339278 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.339259 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 09:18:46.350978 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.350961 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sgnqf"] Jul 22 09:18:46.365696 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.365681 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 09:18:46.426531 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.426510 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/800290ab-f0d6-4e93-82a8-5eea0b266100-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hntwv\" (UID: \"800290ab-f0d6-4e93-82a8-5eea0b266100\") " pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.426641 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.426554 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb9wz\" (UniqueName: \"kubernetes.io/projected/800290ab-f0d6-4e93-82a8-5eea0b266100-kube-api-access-wb9wz\") pod \"insights-runtime-extractor-hntwv\" (UID: \"800290ab-f0d6-4e93-82a8-5eea0b266100\") " pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.426641 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.426572 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/800290ab-f0d6-4e93-82a8-5eea0b266100-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hntwv\" (UID: \"800290ab-f0d6-4e93-82a8-5eea0b266100\") " pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.426711 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.426654 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/800290ab-f0d6-4e93-82a8-5eea0b266100-crio-socket\") pod \"insights-runtime-extractor-hntwv\" (UID: \"800290ab-f0d6-4e93-82a8-5eea0b266100\") " pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.426711 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.426701 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/800290ab-f0d6-4e93-82a8-5eea0b266100-data-volume\") pod \"insights-runtime-extractor-hntwv\" (UID: \"800290ab-f0d6-4e93-82a8-5eea0b266100\") " pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.434547 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.434524 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pkdz2"] Jul 22 09:18:46.455089 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.455068 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pkdz2" Jul 22 09:18:46.456512 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.456496 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pkdz2"] Jul 22 09:18:46.459074 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.458856 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 09:18:46.459074 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.458858 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z2clq\"" Jul 22 09:18:46.459074 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.458914 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 09:18:46.527828 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.527797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/800290ab-f0d6-4e93-82a8-5eea0b266100-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hntwv\" (UID: \"800290ab-f0d6-4e93-82a8-5eea0b266100\") " pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.527987 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.527849 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb9wz\" (UniqueName: \"kubernetes.io/projected/800290ab-f0d6-4e93-82a8-5eea0b266100-kube-api-access-wb9wz\") pod \"insights-runtime-extractor-hntwv\" (UID: \"800290ab-f0d6-4e93-82a8-5eea0b266100\") " pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.527987 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.527876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/800290ab-f0d6-4e93-82a8-5eea0b266100-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hntwv\" (UID: \"800290ab-f0d6-4e93-82a8-5eea0b266100\") " pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.527987 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.527922 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/800290ab-f0d6-4e93-82a8-5eea0b266100-crio-socket\") pod \"insights-runtime-extractor-hntwv\" (UID: \"800290ab-f0d6-4e93-82a8-5eea0b266100\") " pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.527987 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.527965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2dls\" (UniqueName: \"kubernetes.io/projected/cb2ebe90-04ef-463b-a79d-374233cb36dd-kube-api-access-x2dls\") pod \"ingress-canary-sgnqf\" (UID: \"cb2ebe90-04ef-463b-a79d-374233cb36dd\") " pod="openshift-ingress-canary/ingress-canary-sgnqf" Jul 22 09:18:46.528190 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.528007 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb2ebe90-04ef-463b-a79d-374233cb36dd-cert\") pod \"ingress-canary-sgnqf\" (UID: \"cb2ebe90-04ef-463b-a79d-374233cb36dd\") " pod="openshift-ingress-canary/ingress-canary-sgnqf" Jul 22 09:18:46.528190 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.528042 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/800290ab-f0d6-4e93-82a8-5eea0b266100-data-volume\") pod \"insights-runtime-extractor-hntwv\" (UID: \"800290ab-f0d6-4e93-82a8-5eea0b266100\") " pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.528357 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.528121 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/800290ab-f0d6-4e93-82a8-5eea0b266100-crio-socket\") pod \"insights-runtime-extractor-hntwv\" (UID: \"800290ab-f0d6-4e93-82a8-5eea0b266100\") " pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.528654 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.528635 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/800290ab-f0d6-4e93-82a8-5eea0b266100-data-volume\") pod \"insights-runtime-extractor-hntwv\" (UID: \"800290ab-f0d6-4e93-82a8-5eea0b266100\") " pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.528974 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.528952 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/800290ab-f0d6-4e93-82a8-5eea0b266100-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hntwv\" (UID: \"800290ab-f0d6-4e93-82a8-5eea0b266100\") " pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.532810 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.532790 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/800290ab-f0d6-4e93-82a8-5eea0b266100-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hntwv\" (UID: \"800290ab-f0d6-4e93-82a8-5eea0b266100\") " pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.539319 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.539296 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb9wz\" (UniqueName: \"kubernetes.io/projected/800290ab-f0d6-4e93-82a8-5eea0b266100-kube-api-access-wb9wz\") pod \"insights-runtime-extractor-hntwv\" (UID: \"800290ab-f0d6-4e93-82a8-5eea0b266100\") " pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.628429 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.628357 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb2ebe90-04ef-463b-a79d-374233cb36dd-cert\") pod \"ingress-canary-sgnqf\" (UID: \"cb2ebe90-04ef-463b-a79d-374233cb36dd\") " pod="openshift-ingress-canary/ingress-canary-sgnqf" Jul 22 09:18:46.628429 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.628402 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6106e154-c174-4852-a1c0-52d15d988868-metrics-tls\") pod \"dns-default-pkdz2\" (UID: \"6106e154-c174-4852-a1c0-52d15d988868\") " pod="openshift-dns/dns-default-pkdz2" Jul 22 09:18:46.628576 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.628514 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5qvr\" (UniqueName: \"kubernetes.io/projected/6106e154-c174-4852-a1c0-52d15d988868-kube-api-access-n5qvr\") pod \"dns-default-pkdz2\" (UID: \"6106e154-c174-4852-a1c0-52d15d988868\") " pod="openshift-dns/dns-default-pkdz2" Jul 22 09:18:46.628576 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.628563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6106e154-c174-4852-a1c0-52d15d988868-tmp-dir\") pod \"dns-default-pkdz2\" (UID: \"6106e154-c174-4852-a1c0-52d15d988868\") " pod="openshift-dns/dns-default-pkdz2" Jul 22 09:18:46.628644 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.628617 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6106e154-c174-4852-a1c0-52d15d988868-config-volume\") pod \"dns-default-pkdz2\" (UID: \"6106e154-c174-4852-a1c0-52d15d988868\") " pod="openshift-dns/dns-default-pkdz2" Jul 22 09:18:46.628700 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.628685 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2dls\" (UniqueName: \"kubernetes.io/projected/cb2ebe90-04ef-463b-a79d-374233cb36dd-kube-api-access-x2dls\") pod \"ingress-canary-sgnqf\" (UID: \"cb2ebe90-04ef-463b-a79d-374233cb36dd\") " pod="openshift-ingress-canary/ingress-canary-sgnqf" Jul 22 09:18:46.629922 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.629900 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hntwv" Jul 22 09:18:46.630838 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.630819 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb2ebe90-04ef-463b-a79d-374233cb36dd-cert\") pod \"ingress-canary-sgnqf\" (UID: \"cb2ebe90-04ef-463b-a79d-374233cb36dd\") " pod="openshift-ingress-canary/ingress-canary-sgnqf" Jul 22 09:18:46.641146 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.641128 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2dls\" (UniqueName: \"kubernetes.io/projected/cb2ebe90-04ef-463b-a79d-374233cb36dd-kube-api-access-x2dls\") pod \"ingress-canary-sgnqf\" (UID: \"cb2ebe90-04ef-463b-a79d-374233cb36dd\") " pod="openshift-ingress-canary/ingress-canary-sgnqf" Jul 22 09:18:46.642917 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.642903 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sgnqf" Jul 22 09:18:46.729433 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.729378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6106e154-c174-4852-a1c0-52d15d988868-tmp-dir\") pod \"dns-default-pkdz2\" (UID: \"6106e154-c174-4852-a1c0-52d15d988868\") " pod="openshift-dns/dns-default-pkdz2" Jul 22 09:18:46.729557 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.729442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6106e154-c174-4852-a1c0-52d15d988868-config-volume\") pod \"dns-default-pkdz2\" (UID: \"6106e154-c174-4852-a1c0-52d15d988868\") " pod="openshift-dns/dns-default-pkdz2" Jul 22 09:18:46.729557 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.729525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6106e154-c174-4852-a1c0-52d15d988868-metrics-tls\") pod \"dns-default-pkdz2\" (UID: \"6106e154-c174-4852-a1c0-52d15d988868\") " pod="openshift-dns/dns-default-pkdz2" Jul 22 09:18:46.729557 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.729552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5qvr\" (UniqueName: \"kubernetes.io/projected/6106e154-c174-4852-a1c0-52d15d988868-kube-api-access-n5qvr\") pod \"dns-default-pkdz2\" (UID: \"6106e154-c174-4852-a1c0-52d15d988868\") " pod="openshift-dns/dns-default-pkdz2" Jul 22 09:18:46.730580 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.730397 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6106e154-c174-4852-a1c0-52d15d988868-config-volume\") pod \"dns-default-pkdz2\" (UID: \"6106e154-c174-4852-a1c0-52d15d988868\") " pod="openshift-dns/dns-default-pkdz2" Jul 22 09:18:46.730862 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.730805 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6106e154-c174-4852-a1c0-52d15d988868-tmp-dir\") pod \"dns-default-pkdz2\" (UID: \"6106e154-c174-4852-a1c0-52d15d988868\") " pod="openshift-dns/dns-default-pkdz2" Jul 22 09:18:46.733270 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.733244 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6106e154-c174-4852-a1c0-52d15d988868-metrics-tls\") pod \"dns-default-pkdz2\" (UID: \"6106e154-c174-4852-a1c0-52d15d988868\") " pod="openshift-dns/dns-default-pkdz2" Jul 22 09:18:46.738110 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.738086 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5qvr\" (UniqueName: \"kubernetes.io/projected/6106e154-c174-4852-a1c0-52d15d988868-kube-api-access-n5qvr\") pod \"dns-default-pkdz2\" (UID: \"6106e154-c174-4852-a1c0-52d15d988868\") " pod="openshift-dns/dns-default-pkdz2" Jul 22 09:18:46.764267 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.764236 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pkdz2" Jul 22 09:18:46.803374 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.801553 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hntwv"] Jul 22 09:18:46.812996 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:18:46.812967 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800290ab_f0d6_4e93_82a8_5eea0b266100.slice/crio-9e73cda78dd4ffbf02a3aa0a7865cd1b7295f3b4a1cbcde9ba6786a5d1621584 WatchSource:0}: Error finding container 9e73cda78dd4ffbf02a3aa0a7865cd1b7295f3b4a1cbcde9ba6786a5d1621584: Status 404 returned error can't find the container with id 9e73cda78dd4ffbf02a3aa0a7865cd1b7295f3b4a1cbcde9ba6786a5d1621584 Jul 22 09:18:46.829826 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.829798 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sgnqf"] Jul 22 09:18:46.832568 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:18:46.832542 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2ebe90_04ef_463b_a79d_374233cb36dd.slice/crio-c6a34fb37d7a2b8cacfde123a5a3663608562df3c6d7f6297ec11914474e1e26 WatchSource:0}: Error finding container c6a34fb37d7a2b8cacfde123a5a3663608562df3c6d7f6297ec11914474e1e26: Status 404 returned error can't find the container with id c6a34fb37d7a2b8cacfde123a5a3663608562df3c6d7f6297ec11914474e1e26 Jul 22 09:18:46.913540 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:46.913471 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pkdz2"] Jul 22 09:18:46.916232 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:18:46.916206 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6106e154_c174_4852_a1c0_52d15d988868.slice/crio-b7eb93ab884195e8d2e21318bc143320530fac2b7b3c04b2aaa2bf904e02d6d1 WatchSource:0}: Error finding container b7eb93ab884195e8d2e21318bc143320530fac2b7b3c04b2aaa2bf904e02d6d1: Status 404 returned error can't find the container with id b7eb93ab884195e8d2e21318bc143320530fac2b7b3c04b2aaa2bf904e02d6d1 Jul 22 09:18:47.008749 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:47.008718 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:18:47.008905 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:47.008718 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:18:47.013442 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:47.013424 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 09:18:47.013821 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:47.013539 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 09:18:47.013821 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:47.013547 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 09:18:47.014428 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:47.014412 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8b4k6\"" Jul 22 09:18:47.205535 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:47.205500 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pkdz2" event={"ID":"6106e154-c174-4852-a1c0-52d15d988868","Type":"ContainerStarted","Data":"b7eb93ab884195e8d2e21318bc143320530fac2b7b3c04b2aaa2bf904e02d6d1"} Jul 22 09:18:47.206798 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:47.206769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sgnqf" event={"ID":"cb2ebe90-04ef-463b-a79d-374233cb36dd","Type":"ContainerStarted","Data":"c6a34fb37d7a2b8cacfde123a5a3663608562df3c6d7f6297ec11914474e1e26"} Jul 22 09:18:47.208003 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:47.207979 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hntwv" event={"ID":"800290ab-f0d6-4e93-82a8-5eea0b266100","Type":"ContainerStarted","Data":"9e73cda78dd4ffbf02a3aa0a7865cd1b7295f3b4a1cbcde9ba6786a5d1621584"} Jul 22 09:18:48.008641 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:48.008540 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:18:48.014413 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:48.014008 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6sgn5\"" Jul 22 09:18:48.014413 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:48.014261 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 09:18:53.228551 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:53.228165 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pkdz2" event={"ID":"6106e154-c174-4852-a1c0-52d15d988868","Type":"ContainerStarted","Data":"13881cffd9d915947ff7a4e79a8128b7c645bbef5d475f09d9e909caeddd6819"} Jul 22 09:18:53.228551 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:53.228261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pkdz2" event={"ID":"6106e154-c174-4852-a1c0-52d15d988868","Type":"ContainerStarted","Data":"47a3fb983993e63eb362da25497d4c22a40a6cd27b49144a639fba0f3c75e420"} Jul 22 09:18:53.228551 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:53.228460 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pkdz2" Jul 22 09:18:53.230498 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:53.230468 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sgnqf" event={"ID":"cb2ebe90-04ef-463b-a79d-374233cb36dd","Type":"ContainerStarted","Data":"f831c392f3fb36b31f90e3212b7a6f6c902f7faa9af15a69a73f08f55bfab233"} Jul 22 09:18:53.232690 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:53.232630 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hntwv" event={"ID":"800290ab-f0d6-4e93-82a8-5eea0b266100","Type":"ContainerStarted","Data":"0512cc843550c95cbb3a61b44566cb59462ac97f65e69b4200454739e452f33b"} Jul 22 09:18:53.249591 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:53.249537 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pkdz2" podStartSLOduration=1.746814997 podStartE2EDuration="7.249520341s" totalImagesPullingTime="5.502705344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:53.249096845 +0000 UTC m=+57.746604282" watchObservedRunningTime="2026-07-22 09:18:53.249520341 +0000 UTC m=+57.747027760" Jul 22 09:18:53.266596 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:53.266552 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sgnqf" podStartSLOduration=1.682405806 podStartE2EDuration="7.266537272s" totalImagesPullingTime="5.584131466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:53.26548605 +0000 UTC m=+57.762993487" watchObservedRunningTime="2026-07-22 09:18:53.266537272 +0000 UTC m=+57.764044750" Jul 22 09:18:55.500029 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:55.499962 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4hmnr"] Jul 22 09:18:55.505275 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:55.505247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4hmnr" Jul 22 09:18:55.507777 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:55.507750 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 22 09:18:55.507913 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:55.507821 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-x97gr\"" Jul 22 09:18:55.512095 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:55.512039 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4hmnr"] Jul 22 09:18:55.591246 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:55.591212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/73cdba8d-82d0-4c79-a4f1-dea7c3fa8036-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-4hmnr\" (UID: \"73cdba8d-82d0-4c79-a4f1-dea7c3fa8036\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4hmnr" Jul 22 09:18:55.691996 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:55.691959 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/73cdba8d-82d0-4c79-a4f1-dea7c3fa8036-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-4hmnr\" (UID: \"73cdba8d-82d0-4c79-a4f1-dea7c3fa8036\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4hmnr" Jul 22 09:18:55.694691 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:55.694667 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/73cdba8d-82d0-4c79-a4f1-dea7c3fa8036-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-4hmnr\" (UID: \"73cdba8d-82d0-4c79-a4f1-dea7c3fa8036\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4hmnr" Jul 22 09:18:55.815836 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:55.815753 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4hmnr" Jul 22 09:18:56.096512 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:56.096449 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4hmnr"] Jul 22 09:18:56.099210 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:18:56.099187 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73cdba8d_82d0_4c79_a4f1_dea7c3fa8036.slice/crio-ea50517f0a9e9c659534fd66bec6292ee8b4531f1b9bdf065f13f3639b493150 WatchSource:0}: Error finding container ea50517f0a9e9c659534fd66bec6292ee8b4531f1b9bdf065f13f3639b493150: Status 404 returned error can't find the container with id ea50517f0a9e9c659534fd66bec6292ee8b4531f1b9bdf065f13f3639b493150 Jul 22 09:18:56.160760 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:56.160734 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mdr8h" Jul 22 09:18:56.241749 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:56.241717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hntwv" event={"ID":"800290ab-f0d6-4e93-82a8-5eea0b266100","Type":"ContainerStarted","Data":"2cc2f5b3b985c12e7c833c58ad214e814584fd8ef7c8df3817e443e033ce6ea9"} Jul 22 09:18:56.242835 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:56.242811 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4hmnr" event={"ID":"73cdba8d-82d0-4c79-a4f1-dea7c3fa8036","Type":"ContainerStarted","Data":"ea50517f0a9e9c659534fd66bec6292ee8b4531f1b9bdf065f13f3639b493150"} Jul 22 09:18:57.248567 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.248528 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4hmnr" event={"ID":"73cdba8d-82d0-4c79-a4f1-dea7c3fa8036","Type":"ContainerStarted","Data":"60ead9c41590cb06575966d39ad8a82f76bbf2240687a64da3df0f9745ff4a55"} Jul 22 09:18:57.249026 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.248752 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4hmnr" Jul 22 09:18:57.254626 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.254603 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4hmnr" Jul 22 09:18:57.266417 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.266377 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-4hmnr" podStartSLOduration=1.268245775 podStartE2EDuration="2.266364763s" totalImagesPullingTime="998.118988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:18:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:18:57.264788717 +0000 UTC m=+61.762296156" watchObservedRunningTime="2026-07-22 09:18:57.266364763 +0000 UTC m=+61.763872225" Jul 22 09:18:57.574233 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.574197 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx"] Jul 22 09:18:57.576377 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.576357 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" Jul 22 09:18:57.581281 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.581253 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 09:18:57.581392 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.581288 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 22 09:18:57.581392 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.581252 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 09:18:57.582426 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.582303 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7mxg6\"" Jul 22 09:18:57.582426 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.582390 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 22 09:18:57.582611 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.582597 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 09:18:57.602471 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.602447 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx"] Jul 22 09:18:57.605754 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.605720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb89b\" (UniqueName: \"kubernetes.io/projected/3a5806ce-e8e1-4a25-8bfc-15f559539bb1-kube-api-access-kb89b\") pod \"prometheus-operator-5945b5cfd-6zrzx\" (UID: \"3a5806ce-e8e1-4a25-8bfc-15f559539bb1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" Jul 22 09:18:57.605851 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.605774 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a5806ce-e8e1-4a25-8bfc-15f559539bb1-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-6zrzx\" (UID: \"3a5806ce-e8e1-4a25-8bfc-15f559539bb1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" Jul 22 09:18:57.605907 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.605871 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a5806ce-e8e1-4a25-8bfc-15f559539bb1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-6zrzx\" (UID: \"3a5806ce-e8e1-4a25-8bfc-15f559539bb1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" Jul 22 09:18:57.605974 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.605962 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a5806ce-e8e1-4a25-8bfc-15f559539bb1-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6zrzx\" (UID: \"3a5806ce-e8e1-4a25-8bfc-15f559539bb1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" Jul 22 09:18:57.706710 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.706670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a5806ce-e8e1-4a25-8bfc-15f559539bb1-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6zrzx\" (UID: \"3a5806ce-e8e1-4a25-8bfc-15f559539bb1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" Jul 22 09:18:57.706892 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.706725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb89b\" (UniqueName: \"kubernetes.io/projected/3a5806ce-e8e1-4a25-8bfc-15f559539bb1-kube-api-access-kb89b\") pod \"prometheus-operator-5945b5cfd-6zrzx\" (UID: \"3a5806ce-e8e1-4a25-8bfc-15f559539bb1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" Jul 22 09:18:57.706892 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.706767 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a5806ce-e8e1-4a25-8bfc-15f559539bb1-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-6zrzx\" (UID: \"3a5806ce-e8e1-4a25-8bfc-15f559539bb1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" Jul 22 09:18:57.706892 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.706821 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a5806ce-e8e1-4a25-8bfc-15f559539bb1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-6zrzx\" (UID: \"3a5806ce-e8e1-4a25-8bfc-15f559539bb1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" Jul 22 09:18:57.706892 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:57.706837 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 22 09:18:57.707126 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:18:57.706917 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a5806ce-e8e1-4a25-8bfc-15f559539bb1-prometheus-operator-tls podName:3a5806ce-e8e1-4a25-8bfc-15f559539bb1 nodeName:}" failed. No retries permitted until 2026-07-22 09:18:58.206897035 +0000 UTC m=+62.704404450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/3a5806ce-e8e1-4a25-8bfc-15f559539bb1-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-6zrzx" (UID: "3a5806ce-e8e1-4a25-8bfc-15f559539bb1") : secret "prometheus-operator-tls" not found Jul 22 09:18:57.707551 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.707520 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a5806ce-e8e1-4a25-8bfc-15f559539bb1-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-6zrzx\" (UID: \"3a5806ce-e8e1-4a25-8bfc-15f559539bb1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" Jul 22 09:18:57.709448 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.709422 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a5806ce-e8e1-4a25-8bfc-15f559539bb1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-6zrzx\" (UID: \"3a5806ce-e8e1-4a25-8bfc-15f559539bb1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" Jul 22 09:18:57.716174 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.716154 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb89b\" (UniqueName: \"kubernetes.io/projected/3a5806ce-e8e1-4a25-8bfc-15f559539bb1-kube-api-access-kb89b\") pod \"prometheus-operator-5945b5cfd-6zrzx\" (UID: \"3a5806ce-e8e1-4a25-8bfc-15f559539bb1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" Jul 22 09:18:57.904962 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.904869 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bd5cff79c-tnkmm"] Jul 22 09:18:57.907763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.907745 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:57.910679 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.910624 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 09:18:57.910807 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.910698 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 09:18:57.910807 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.910697 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 09:18:57.910807 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.910697 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 09:18:57.911084 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.911067 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 09:18:57.911166 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.911069 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jbg2j\"" Jul 22 09:18:57.911678 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.911661 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 09:18:57.911678 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.911674 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 09:18:57.921921 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:57.921902 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bd5cff79c-tnkmm"] Jul 22 09:18:58.009433 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.009281 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-config\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.009433 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.009333 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn5ht\" (UniqueName: \"kubernetes.io/projected/7de0edbf-64ae-41fa-9fce-ca32ab28a277-kube-api-access-zn5ht\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.009433 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.009387 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-serving-cert\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.009433 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.009442 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-oauth-config\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.009739 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.009462 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-oauth-serving-cert\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.009739 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.009478 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-service-ca\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.110322 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.110284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn5ht\" (UniqueName: \"kubernetes.io/projected/7de0edbf-64ae-41fa-9fce-ca32ab28a277-kube-api-access-zn5ht\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.110508 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.110363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-serving-cert\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.110508 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.110444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-oauth-config\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.110508 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.110468 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-oauth-serving-cert\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.110508 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.110497 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-service-ca\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.111229 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.110538 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-config\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.111439 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.111374 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-service-ca\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.111439 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.111413 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-config\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.111654 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.111444 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-oauth-serving-cert\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.113539 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.113519 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-oauth-config\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.113782 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.113758 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-serving-cert\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.120056 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.120013 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn5ht\" (UniqueName: \"kubernetes.io/projected/7de0edbf-64ae-41fa-9fce-ca32ab28a277-kube-api-access-zn5ht\") pod \"console-7bd5cff79c-tnkmm\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.211838 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.211798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a5806ce-e8e1-4a25-8bfc-15f559539bb1-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6zrzx\" (UID: \"3a5806ce-e8e1-4a25-8bfc-15f559539bb1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" Jul 22 09:18:58.214518 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.214487 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a5806ce-e8e1-4a25-8bfc-15f559539bb1-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-6zrzx\" (UID: \"3a5806ce-e8e1-4a25-8bfc-15f559539bb1\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" Jul 22 09:18:58.218349 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.218289 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:18:58.486900 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:58.486824 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" Jul 22 09:18:59.556332 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:59.556301 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bd5cff79c-tnkmm"] Jul 22 09:18:59.559886 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:18:59.559863 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de0edbf_64ae_41fa_9fce_ca32ab28a277.slice/crio-3269bc9bd0a9f72570d0717f558cab89d1e246573a1842b334fc751f03c856e7 WatchSource:0}: Error finding container 3269bc9bd0a9f72570d0717f558cab89d1e246573a1842b334fc751f03c856e7: Status 404 returned error can't find the container with id 3269bc9bd0a9f72570d0717f558cab89d1e246573a1842b334fc751f03c856e7 Jul 22 09:18:59.572163 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:18:59.572140 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx"] Jul 22 09:18:59.575392 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:18:59.575370 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5806ce_e8e1_4a25_8bfc_15f559539bb1.slice/crio-3716f381cf82df1f6b85bb978988d5018ee48587b6d8cb6dbd1ef36a39d79517 WatchSource:0}: Error finding container 3716f381cf82df1f6b85bb978988d5018ee48587b6d8cb6dbd1ef36a39d79517: Status 404 returned error can't find the container with id 3716f381cf82df1f6b85bb978988d5018ee48587b6d8cb6dbd1ef36a39d79517 Jul 22 09:19:00.263732 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:00.263695 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hntwv" event={"ID":"800290ab-f0d6-4e93-82a8-5eea0b266100","Type":"ContainerStarted","Data":"8664cc5b5bca31afb7085b5a6eeabed9140f3edce95137bce089199d9d241727"} Jul 22 09:19:00.264891 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:00.264864 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd5cff79c-tnkmm" event={"ID":"7de0edbf-64ae-41fa-9fce-ca32ab28a277","Type":"ContainerStarted","Data":"3269bc9bd0a9f72570d0717f558cab89d1e246573a1842b334fc751f03c856e7"} Jul 22 09:19:00.266004 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:00.265979 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" event={"ID":"3a5806ce-e8e1-4a25-8bfc-15f559539bb1","Type":"ContainerStarted","Data":"3716f381cf82df1f6b85bb978988d5018ee48587b6d8cb6dbd1ef36a39d79517"} Jul 22 09:19:00.285697 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:00.285654 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hntwv" podStartSLOduration=1.792058573 podStartE2EDuration="14.285641988s" totalImagesPullingTime="12.493583415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:18:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:00.284272887 +0000 UTC m=+64.781780322" watchObservedRunningTime="2026-07-22 09:19:00.285641988 +0000 UTC m=+64.783149423" Jul 22 09:19:01.271199 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.271155 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" event={"ID":"3a5806ce-e8e1-4a25-8bfc-15f559539bb1","Type":"ContainerStarted","Data":"56cdeda4180ab4449232762af7f22b738dd497c96471e7cfe21fe5d2fa605583"} Jul 22 09:19:01.271564 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.271223 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" event={"ID":"3a5806ce-e8e1-4a25-8bfc-15f559539bb1","Type":"ContainerStarted","Data":"a7e62c3af27467b76788f4e091a5cef0f911e941cb073f4dae7d319699ec42af"} Jul 22 09:19:01.290372 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.290299 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-6zrzx" podStartSLOduration=2.940737816 podStartE2EDuration="4.290283273s" totalImagesPullingTime="1.349545457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:01.289509587 +0000 UTC m=+65.787017027" watchObservedRunningTime="2026-07-22 09:19:01.290283273 +0000 UTC m=+65.787790708" Jul 22 09:19:01.639486 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.639396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74lfl\" (UniqueName: \"kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl\") pod \"network-check-target-km7cq\" (UID: \"ec80e86d-c05c-439a-ad34-c44e4009f060\") " pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:19:01.641890 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.641872 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 09:19:01.652399 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.652382 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 09:19:01.662556 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.662532 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74lfl\" (UniqueName: \"kubernetes.io/projected/ec80e86d-c05c-439a-ad34-c44e4009f060-kube-api-access-74lfl\") pod \"network-check-target-km7cq\" (UID: \"ec80e86d-c05c-439a-ad34-c44e4009f060\") " pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:19:01.726084 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.726042 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8b4k6\"" Jul 22 09:19:01.734603 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.734580 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:19:01.740684 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.740664 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs\") pod \"network-metrics-daemon-2wqk7\" (UID: \"e460fb8e-afe6-4628-b08e-514c648282c3\") " pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:19:01.740800 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.740776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:19:01.743141 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.743117 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 09:19:01.743253 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.743206 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 09:19:01.753354 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.753317 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e460fb8e-afe6-4628-b08e-514c648282c3-metrics-certs\") pod \"network-metrics-daemon-2wqk7\" (UID: \"e460fb8e-afe6-4628-b08e-514c648282c3\") " pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:19:01.753470 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.753383 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21a0247b-c2c6-4f3e-a619-2eaaf17a2a06-original-pull-secret\") pod \"global-pull-secret-syncer-5zw6v\" (UID: \"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06\") " pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:19:01.823701 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.823670 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6sgn5\"" Jul 22 09:19:01.831767 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.831744 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2wqk7" Jul 22 09:19:01.852656 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.852636 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-km7cq"] Jul 22 09:19:01.855466 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:19:01.855420 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec80e86d_c05c_439a_ad34_c44e4009f060.slice/crio-aeb5646d95d5f305710cc71597a3da97e3def873282bf6d1e441ae01a39e4ed9 WatchSource:0}: Error finding container aeb5646d95d5f305710cc71597a3da97e3def873282bf6d1e441ae01a39e4ed9: Status 404 returned error can't find the container with id aeb5646d95d5f305710cc71597a3da97e3def873282bf6d1e441ae01a39e4ed9 Jul 22 09:19:01.953124 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:01.953094 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2wqk7"] Jul 22 09:19:01.956190 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:19:01.956167 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode460fb8e_afe6_4628_b08e_514c648282c3.slice/crio-6762b98fffe45e5da808829d2a85e48d7c28e70a3ba1fa3906a495dfe5ec54c6 WatchSource:0}: Error finding container 6762b98fffe45e5da808829d2a85e48d7c28e70a3ba1fa3906a495dfe5ec54c6: Status 404 returned error can't find the container with id 6762b98fffe45e5da808829d2a85e48d7c28e70a3ba1fa3906a495dfe5ec54c6 Jul 22 09:19:02.018300 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.018273 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5zw6v" Jul 22 09:19:02.146361 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.146327 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5zw6v"] Jul 22 09:19:02.149450 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:19:02.149423 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a0247b_c2c6_4f3e_a619_2eaaf17a2a06.slice/crio-57d690960c7e059b25f3bbac124ee5bd1bc658465b4c17e4319da1157c93e189 WatchSource:0}: Error finding container 57d690960c7e059b25f3bbac124ee5bd1bc658465b4c17e4319da1157c93e189: Status 404 returned error can't find the container with id 57d690960c7e059b25f3bbac124ee5bd1bc658465b4c17e4319da1157c93e189 Jul 22 09:19:02.276681 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.276637 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5zw6v" event={"ID":"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06","Type":"ContainerStarted","Data":"57d690960c7e059b25f3bbac124ee5bd1bc658465b4c17e4319da1157c93e189"} Jul 22 09:19:02.277967 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.277921 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2wqk7" event={"ID":"e460fb8e-afe6-4628-b08e-514c648282c3","Type":"ContainerStarted","Data":"6762b98fffe45e5da808829d2a85e48d7c28e70a3ba1fa3906a495dfe5ec54c6"} Jul 22 09:19:02.278966 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.278926 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-km7cq" event={"ID":"ec80e86d-c05c-439a-ad34-c44e4009f060","Type":"ContainerStarted","Data":"aeb5646d95d5f305710cc71597a3da97e3def873282bf6d1e441ae01a39e4ed9"} Jul 22 09:19:02.916964 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.916840 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds"] Jul 22 09:19:02.919812 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.919786 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" Jul 22 09:19:02.922482 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.922449 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 22 09:19:02.922607 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.922506 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 22 09:19:02.922685 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.922675 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cj4d7\"" Jul 22 09:19:02.927309 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.927280 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-xbttw"] Jul 22 09:19:02.938188 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.937284 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds"] Jul 22 09:19:02.938188 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.937428 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:02.940812 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.940421 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vcmnd\"" Jul 22 09:19:02.940812 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.940516 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 22 09:19:02.940812 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.940690 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 22 09:19:02.941061 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.940814 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 22 09:19:02.943684 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.943302 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-xbttw"] Jul 22 09:19:02.946912 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.946887 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fc79w"] Jul 22 09:19:02.950069 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.949701 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21d042ab-2267-4459-9215-01e50bf7eee8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-rbpds\" (UID: \"21d042ab-2267-4459-9215-01e50bf7eee8\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" Jul 22 09:19:02.950069 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.949770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21d042ab-2267-4459-9215-01e50bf7eee8-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-rbpds\" (UID: \"21d042ab-2267-4459-9215-01e50bf7eee8\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" Jul 22 09:19:02.950069 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.949803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mm4r\" (UniqueName: \"kubernetes.io/projected/21d042ab-2267-4459-9215-01e50bf7eee8-kube-api-access-5mm4r\") pod \"openshift-state-metrics-79f6958cdf-rbpds\" (UID: \"21d042ab-2267-4459-9215-01e50bf7eee8\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" Jul 22 09:19:02.950069 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.949852 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/21d042ab-2267-4459-9215-01e50bf7eee8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-rbpds\" (UID: \"21d042ab-2267-4459-9215-01e50bf7eee8\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" Jul 22 09:19:02.950846 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.950827 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:02.953230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.953207 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 09:19:02.953343 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.953286 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 09:19:02.953519 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.953500 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 09:19:02.953664 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:02.953645 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jgbts\"" Jul 22 09:19:03.051113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21d042ab-2267-4459-9215-01e50bf7eee8-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-rbpds\" (UID: \"21d042ab-2267-4459-9215-01e50bf7eee8\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" Jul 22 09:19:03.051113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.051113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhmvf\" (UniqueName: \"kubernetes.io/projected/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-kube-api-access-dhmvf\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.051113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050438 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-node-exporter-accelerators-collector-config\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.051113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050476 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-metrics-client-ca\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.051113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050494 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.051113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050533 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-root\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.051113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/21d042ab-2267-4459-9215-01e50bf7eee8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-rbpds\" (UID: \"21d042ab-2267-4459-9215-01e50bf7eee8\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" Jul 22 09:19:03.051113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050619 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-node-exporter-wtmp\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.051113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050653 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqqgd\" (UniqueName: \"kubernetes.io/projected/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-kube-api-access-rqqgd\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.051113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21d042ab-2267-4459-9215-01e50bf7eee8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-rbpds\" (UID: \"21d042ab-2267-4459-9215-01e50bf7eee8\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" Jul 22 09:19:03.051113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.051113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.051113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050774 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.051113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mm4r\" (UniqueName: \"kubernetes.io/projected/21d042ab-2267-4459-9215-01e50bf7eee8-kube-api-access-5mm4r\") pod \"openshift-state-metrics-79f6958cdf-rbpds\" (UID: \"21d042ab-2267-4459-9215-01e50bf7eee8\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" Jul 22 09:19:03.052145 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050845 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-node-exporter-textfile\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.052145 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.052145 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050904 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-node-exporter-tls\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.052145 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.050930 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-sys\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.052145 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.051648 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21d042ab-2267-4459-9215-01e50bf7eee8-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-rbpds\" (UID: \"21d042ab-2267-4459-9215-01e50bf7eee8\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" Jul 22 09:19:03.054471 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.054444 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/21d042ab-2267-4459-9215-01e50bf7eee8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-rbpds\" (UID: \"21d042ab-2267-4459-9215-01e50bf7eee8\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" Jul 22 09:19:03.054870 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.054819 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/21d042ab-2267-4459-9215-01e50bf7eee8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-rbpds\" (UID: \"21d042ab-2267-4459-9215-01e50bf7eee8\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" Jul 22 09:19:03.064583 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.064555 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mm4r\" (UniqueName: \"kubernetes.io/projected/21d042ab-2267-4459-9215-01e50bf7eee8-kube-api-access-5mm4r\") pod \"openshift-state-metrics-79f6958cdf-rbpds\" (UID: \"21d042ab-2267-4459-9215-01e50bf7eee8\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" Jul 22 09:19:03.153250 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.152627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-node-exporter-wtmp\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.153250 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.152671 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqqgd\" (UniqueName: \"kubernetes.io/projected/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-kube-api-access-rqqgd\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.153250 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.152708 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.153250 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.152739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.153250 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.152764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.153250 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.152802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-node-exporter-textfile\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.153250 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.152834 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.153250 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.152861 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-node-exporter-tls\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.155214 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.153924 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-sys\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.155214 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.154137 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.155214 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.154171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhmvf\" (UniqueName: \"kubernetes.io/projected/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-kube-api-access-dhmvf\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.155214 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.154197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-node-exporter-accelerators-collector-config\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.155214 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.154228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-metrics-client-ca\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.155214 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.154252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.155214 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.154283 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-root\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.155214 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.154382 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-root\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.155825 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.155393 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.155825 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.155609 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-node-exporter-wtmp\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.155988 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.155880 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-sys\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.156996 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.156596 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.157857 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.157719 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-metrics-client-ca\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.158521 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.158244 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-node-exporter-accelerators-collector-config\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.159989 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.159020 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-node-exporter-textfile\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.159989 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:19:03.159118 2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 22 09:19:03.159989 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:19:03.159178 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-kube-state-metrics-tls podName:2ad4272f-1391-4472-a944-c3bc4f8ddfb5 nodeName:}" failed. No retries permitted until 2026-07-22 09:19:03.659159407 +0000 UTC m=+68.156666823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-xbttw" (UID: "2ad4272f-1391-4472-a944-c3bc4f8ddfb5") : secret "kube-state-metrics-tls" not found Jul 22 09:19:03.159989 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.159914 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.160786 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.160764 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.161781 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.161735 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-node-exporter-tls\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.163448 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.163407 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.168171 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.168073 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhmvf\" (UniqueName: \"kubernetes.io/projected/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-kube-api-access-dhmvf\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.169619 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.169599 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqqgd\" (UniqueName: \"kubernetes.io/projected/9730e0f1-91ed-44fa-ae9f-df4e8ed99a15-kube-api-access-rqqgd\") pod \"node-exporter-fc79w\" (UID: \"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15\") " pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.237664 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.237631 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pkdz2" Jul 22 09:19:03.242525 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.242497 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" Jul 22 09:19:03.264057 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.264028 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fc79w" Jul 22 09:19:03.292862 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.292501 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2wqk7" event={"ID":"e460fb8e-afe6-4628-b08e-514c648282c3","Type":"ContainerStarted","Data":"9c64847852b89befb8c1f05d611831ec31d575b7f0d7a9408eca9c2b98cefed8"} Jul 22 09:19:03.292862 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.292567 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2wqk7" event={"ID":"e460fb8e-afe6-4628-b08e-514c648282c3","Type":"ContainerStarted","Data":"4477f97b9202051c16a383384f31258e2b88699b2de2d3da9c152b751b7fdde8"} Jul 22 09:19:03.423018 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.422890 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2wqk7" podStartSLOduration=66.404898157 podStartE2EDuration="1m7.422866714s" totalImagesPullingTime="1.017968557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:03.311789577 +0000 UTC m=+67.809297015" watchObservedRunningTime="2026-07-22 09:19:03.422866714 +0000 UTC m=+67.920374151" Jul 22 09:19:03.423553 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.423528 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds"] Jul 22 09:19:03.427496 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:19:03.427459 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d042ab_2267_4459_9215_01e50bf7eee8.slice/crio-542b68bf0b44dfc4414b77d7ba53b99fdc2e39fc380ec5303b80b4046c93eb8e WatchSource:0}: Error finding container 542b68bf0b44dfc4414b77d7ba53b99fdc2e39fc380ec5303b80b4046c93eb8e: Status 404 returned error can't find the container with id 542b68bf0b44dfc4414b77d7ba53b99fdc2e39fc380ec5303b80b4046c93eb8e Jul 22 09:19:03.719171 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.717878 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-998d7989b-d5w8r"] Jul 22 09:19:03.721085 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.720494 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.730966 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.730911 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 09:19:03.735778 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.735717 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-998d7989b-d5w8r"] Jul 22 09:19:03.759776 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.759692 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-service-ca\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.759776 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.759739 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-serving-cert\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.760019 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.759784 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-trusted-ca-bundle\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.760019 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.759860 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-config\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.760019 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.759891 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9l4f\" (UniqueName: \"kubernetes.io/projected/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-kube-api-access-w9l4f\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.760019 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.759930 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.760019 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.759993 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-oauth-config\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.760251 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.760031 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-oauth-serving-cert\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.763539 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.763511 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ad4272f-1391-4472-a944-c3bc4f8ddfb5-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-xbttw\" (UID: \"2ad4272f-1391-4472-a944-c3bc4f8ddfb5\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.855093 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.855032 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" Jul 22 09:19:03.860838 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.860801 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-config\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.860962 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.860852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9l4f\" (UniqueName: \"kubernetes.io/projected/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-kube-api-access-w9l4f\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.860962 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.860894 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-oauth-config\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.860962 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.860928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-oauth-serving-cert\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.861134 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.860986 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-service-ca\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.861134 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.861018 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-serving-cert\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.861134 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.861049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-trusted-ca-bundle\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.862728 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.862676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-config\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.871269 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.871146 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-service-ca\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.873155 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.873057 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-trusted-ca-bundle\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.874021 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.873891 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-oauth-serving-cert\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.876726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.876679 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-oauth-config\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.878009 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.877976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9l4f\" (UniqueName: \"kubernetes.io/projected/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-kube-api-access-w9l4f\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:03.879035 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:03.878994 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-serving-cert\") pod \"console-998d7989b-d5w8r\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:04.034438 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.034403 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:04.062648 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.062600 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-xbttw"] Jul 22 09:19:04.064861 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.064601 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 09:19:04.067893 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:19:04.067864 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad4272f_1391_4472_a944_c3bc4f8ddfb5.slice/crio-1aed7bc516a3f7f6bf948961196a181b0b180d42a220576eec403f2159137d69 WatchSource:0}: Error finding container 1aed7bc516a3f7f6bf948961196a181b0b180d42a220576eec403f2159137d69: Status 404 returned error can't find the container with id 1aed7bc516a3f7f6bf948961196a181b0b180d42a220576eec403f2159137d69 Jul 22 09:19:04.069604 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.069584 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.082324 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.082303 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 09:19:04.082649 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.082623 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 09:19:04.082804 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.082785 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 09:19:04.083000 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.082984 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 09:19:04.083103 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.083086 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 09:19:04.083368 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.083310 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wwzkn\"" Jul 22 09:19:04.083488 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.083474 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 09:19:04.084188 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.084144 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 09:19:04.084392 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.084376 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 09:19:04.089402 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.089216 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 09:19:04.094551 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.094249 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 09:19:04.170142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.169387 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-config-out\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.170142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.169432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.170142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.169458 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.170142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.169495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.170142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.169521 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-web-config\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.170142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.169549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.170142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.169603 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.170142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.169627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp5jd\" (UniqueName: \"kubernetes.io/projected/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-kube-api-access-pp5jd\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.170142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.169653 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.170142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.169675 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.170142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.169708 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.170142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.169741 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.170142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.169766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.240691 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.240561 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-998d7989b-d5w8r"] Jul 22 09:19:04.247044 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:19:04.247009 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c6c23c_8c3a_42c3_bd8d_33019807a0ed.slice/crio-5a9e532f188adab1a80f46562d6fd3b8b5778ff991ca3f96dc4c822715d86064 WatchSource:0}: Error finding container 5a9e532f188adab1a80f46562d6fd3b8b5778ff991ca3f96dc4c822715d86064: Status 404 returned error can't find the container with id 5a9e532f188adab1a80f46562d6fd3b8b5778ff991ca3f96dc4c822715d86064 Jul 22 09:19:04.271575 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.271129 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.271575 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.271181 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.271575 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.271232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-config-out\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.271575 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.271258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.271575 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.271283 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.271575 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.271319 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.271575 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.271346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-web-config\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.271575 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.271376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.271575 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.271418 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.271575 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.271441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp5jd\" (UniqueName: \"kubernetes.io/projected/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-kube-api-access-pp5jd\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.271575 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.271466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.271575 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.271488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.271575 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.271521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.272658 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.272306 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.272658 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:19:04.272441 2564 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 22 09:19:04.272658 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:19:04.272502 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-main-tls podName:26e991e9-0547-42ae-8cf8-f98a7fe4b8e2 nodeName:}" failed. No retries permitted until 2026-07-22 09:19:04.772483227 +0000 UTC m=+69.269990641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2") : secret "alertmanager-main-tls" not found Jul 22 09:19:04.272921 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.272871 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.273819 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.273795 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.279039 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.278901 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-config-out\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.280455 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.280353 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.280455 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.280368 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-web-config\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.280913 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.280873 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.282259 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.281629 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.282259 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.282059 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.283530 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.283508 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.290491 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.290179 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.290491 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.290446 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp5jd\" (UniqueName: \"kubernetes.io/projected/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-kube-api-access-pp5jd\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.301154 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.301115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" event={"ID":"2ad4272f-1391-4472-a944-c3bc4f8ddfb5","Type":"ContainerStarted","Data":"1aed7bc516a3f7f6bf948961196a181b0b180d42a220576eec403f2159137d69"} Jul 22 09:19:04.302495 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.302439 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fc79w" event={"ID":"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15","Type":"ContainerStarted","Data":"471a4162cf8f8a852fad94b62e647ebe121e6740a16e62bada6e13e1d4380f11"} Jul 22 09:19:04.306393 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.306362 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" event={"ID":"21d042ab-2267-4459-9215-01e50bf7eee8","Type":"ContainerStarted","Data":"90bb3fc7bb1dc1d406cfcfe52c4c823f7eb03ec71fbf2e90330789c69f4fa4ec"} Jul 22 09:19:04.306491 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.306431 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" event={"ID":"21d042ab-2267-4459-9215-01e50bf7eee8","Type":"ContainerStarted","Data":"8a165a8aa84b074a0c918aa05aed09bba7a21806c05be2748f9b51bf37c000d1"} Jul 22 09:19:04.306491 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.306464 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" event={"ID":"21d042ab-2267-4459-9215-01e50bf7eee8","Type":"ContainerStarted","Data":"542b68bf0b44dfc4414b77d7ba53b99fdc2e39fc380ec5303b80b4046c93eb8e"} Jul 22 09:19:04.308639 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.308414 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-998d7989b-d5w8r" event={"ID":"86c6c23c-8c3a-42c3-bd8d-33019807a0ed","Type":"ContainerStarted","Data":"5a9e532f188adab1a80f46562d6fd3b8b5778ff991ca3f96dc4c822715d86064"} Jul 22 09:19:04.777922 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.777881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.781015 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.780981 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:04.987832 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:04.987550 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:19:06.059726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.059687 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-66c7958d54-c7w2d"] Jul 22 09:19:06.080557 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.080526 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-66c7958d54-c7w2d"] Jul 22 09:19:06.080730 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.080636 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.083773 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.083536 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 22 09:19:06.083773 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.083589 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 22 09:19:06.083773 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.083537 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 22 09:19:06.084142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.084121 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 22 09:19:06.084251 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.084219 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9kr6s5prnmq0k\"" Jul 22 09:19:06.084498 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.084335 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 22 09:19:06.084498 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.084422 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gb4h2\"" Jul 22 09:19:06.191064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.191028 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9mdx\" (UniqueName: \"kubernetes.io/projected/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-kube-api-access-d9mdx\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.191229 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.191092 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.191229 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.191128 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.191229 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.191158 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-thanos-querier-tls\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.191388 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.191272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.191388 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.191306 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-metrics-client-ca\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.191388 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.191324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-grpc-tls\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.191543 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.191400 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.292086 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.292053 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.292234 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.292095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-thanos-querier-tls\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.292234 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.292124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.292234 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.292148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-metrics-client-ca\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.292234 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.292163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-grpc-tls\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.292234 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.292197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.292469 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.292316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9mdx\" (UniqueName: \"kubernetes.io/projected/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-kube-api-access-d9mdx\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.292469 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.292373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.292978 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.292926 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-metrics-client-ca\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.294686 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.294663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.295015 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.294995 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.295148 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.295125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.295207 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.295168 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.295257 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.295224 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-thanos-querier-tls\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.295331 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.295314 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-secret-grpc-tls\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.302430 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.302408 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9mdx\" (UniqueName: \"kubernetes.io/projected/3dae0afa-7c17-4b64-8f0f-fbbf46c632da-kube-api-access-d9mdx\") pod \"thanos-querier-66c7958d54-c7w2d\" (UID: \"3dae0afa-7c17-4b64-8f0f-fbbf46c632da\") " pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:06.393834 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:06.393766 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:07.241795 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.241762 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7fbb65654c-7q7l5"] Jul 22 09:19:07.272317 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.272288 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fbb65654c-7q7l5"] Jul 22 09:19:07.272498 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.272397 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.276603 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.276573 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 22 09:19:07.276738 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.276719 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 22 09:19:07.276803 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.276734 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tnhzg\"" Jul 22 09:19:07.276844 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.276721 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5925f2cm4tqii\"" Jul 22 09:19:07.276902 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.276719 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 22 09:19:07.277038 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.277029 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 22 09:19:07.401082 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.401041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.401082 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.401083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-client-ca-bundle\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.401322 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.401121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-secret-metrics-server-client-certs\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.401322 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.401154 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-secret-metrics-server-tls\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.401322 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.401171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-audit-log\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.401322 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.401251 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgx6v\" (UniqueName: \"kubernetes.io/projected/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-kube-api-access-mgx6v\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.401322 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.401293 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-metrics-server-audit-profiles\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.502529 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.502439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-secret-metrics-server-client-certs\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.502529 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.502509 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-secret-metrics-server-tls\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.502760 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.502538 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-audit-log\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.502760 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.502571 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgx6v\" (UniqueName: \"kubernetes.io/projected/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-kube-api-access-mgx6v\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.502760 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.502600 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-metrics-server-audit-profiles\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.502760 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.502663 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.502760 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.502687 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-client-ca-bundle\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.503346 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.503266 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-audit-log\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.503648 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.503621 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.503899 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.503873 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-metrics-server-audit-profiles\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.505316 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.505292 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-secret-metrics-server-tls\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.505400 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.505378 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-client-ca-bundle\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.505464 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.505442 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-secret-metrics-server-client-certs\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.511419 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.511395 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgx6v\" (UniqueName: \"kubernetes.io/projected/f0832a10-9283-40e6-bb8e-24b6dd30f4e7-kube-api-access-mgx6v\") pod \"metrics-server-7fbb65654c-7q7l5\" (UID: \"f0832a10-9283-40e6-bb8e-24b6dd30f4e7\") " pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.584369 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.584330 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:07.709147 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.709113 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct"] Jul 22 09:19:07.711489 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.711462 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct" Jul 22 09:19:07.714245 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.714222 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 22 09:19:07.714489 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.714465 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lx75f\"" Jul 22 09:19:07.722712 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.722692 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct"] Jul 22 09:19:07.806076 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.805983 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b093f3a3-f6e3-4692-be96-8319e49aea36-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-tqtct\" (UID: \"b093f3a3-f6e3-4692-be96-8319e49aea36\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct" Jul 22 09:19:07.806076 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.806068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49b5\" (UniqueName: \"kubernetes.io/projected/b093f3a3-f6e3-4692-be96-8319e49aea36-kube-api-access-j49b5\") pod \"monitoring-plugin-7b7b7bc8bf-tqtct\" (UID: \"b093f3a3-f6e3-4692-be96-8319e49aea36\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct" Jul 22 09:19:07.907508 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.907472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j49b5\" (UniqueName: \"kubernetes.io/projected/b093f3a3-f6e3-4692-be96-8319e49aea36-kube-api-access-j49b5\") pod \"monitoring-plugin-7b7b7bc8bf-tqtct\" (UID: \"b093f3a3-f6e3-4692-be96-8319e49aea36\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct" Jul 22 09:19:07.907702 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.907584 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b093f3a3-f6e3-4692-be96-8319e49aea36-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-tqtct\" (UID: \"b093f3a3-f6e3-4692-be96-8319e49aea36\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct" Jul 22 09:19:07.907763 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:19:07.907705 2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 22 09:19:07.907814 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:19:07.907777 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b093f3a3-f6e3-4692-be96-8319e49aea36-monitoring-plugin-cert podName:b093f3a3-f6e3-4692-be96-8319e49aea36 nodeName:}" failed. No retries permitted until 2026-07-22 09:19:08.407755424 +0000 UTC m=+72.905262840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/b093f3a3-f6e3-4692-be96-8319e49aea36-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-tqtct" (UID: "b093f3a3-f6e3-4692-be96-8319e49aea36") : secret "monitoring-plugin-cert" not found Jul 22 09:19:07.917336 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:07.917286 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49b5\" (UniqueName: \"kubernetes.io/projected/b093f3a3-f6e3-4692-be96-8319e49aea36-kube-api-access-j49b5\") pod \"monitoring-plugin-7b7b7bc8bf-tqtct\" (UID: \"b093f3a3-f6e3-4692-be96-8319e49aea36\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct" Jul 22 09:19:08.412172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:08.412135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b093f3a3-f6e3-4692-be96-8319e49aea36-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-tqtct\" (UID: \"b093f3a3-f6e3-4692-be96-8319e49aea36\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct" Jul 22 09:19:08.414759 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:08.414732 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b093f3a3-f6e3-4692-be96-8319e49aea36-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-tqtct\" (UID: \"b093f3a3-f6e3-4692-be96-8319e49aea36\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct" Jul 22 09:19:08.623807 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:08.623775 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct" Jul 22 09:19:08.904130 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:08.904097 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bd5cff79c-tnkmm"] Jul 22 09:19:08.938533 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:08.938504 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c486685d6-7gzss"] Jul 22 09:19:08.940701 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:08.940676 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:08.964257 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:08.964228 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c486685d6-7gzss"] Jul 22 09:19:09.018385 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.018335 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-oauth-serving-cert\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.018557 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.018400 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-oauth-config\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.018557 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.018436 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-service-ca\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.018557 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.018488 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csg6m\" (UniqueName: \"kubernetes.io/projected/9edf9893-482b-48a8-9aa0-7baf080b74f1-kube-api-access-csg6m\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.018557 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.018548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-trusted-ca-bundle\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.018756 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.018589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-config\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.018756 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.018632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-serving-cert\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.120328 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.120281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-trusted-ca-bundle\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.120537 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.120342 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-config\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.120537 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.120404 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-serving-cert\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.120537 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.120458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-oauth-serving-cert\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.120537 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.120496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-oauth-config\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.120741 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.120636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-service-ca\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.120741 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.120676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csg6m\" (UniqueName: \"kubernetes.io/projected/9edf9893-482b-48a8-9aa0-7baf080b74f1-kube-api-access-csg6m\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.121395 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.121317 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-trusted-ca-bundle\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.121711 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.121688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-oauth-serving-cert\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.121794 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.121714 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-service-ca\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.122242 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.122204 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-config\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.123575 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.123532 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-oauth-config\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.125324 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.124378 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-serving-cert\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.130149 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.130129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csg6m\" (UniqueName: \"kubernetes.io/projected/9edf9893-482b-48a8-9aa0-7baf080b74f1-kube-api-access-csg6m\") pod \"console-7c486685d6-7gzss\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.253204 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.253173 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 09:19:09.258466 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.258438 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.258641 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.258472 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:09.262815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.261476 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 22 09:19:09.262815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.261492 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zdvcr\"" Jul 22 09:19:09.262815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.261552 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 22 09:19:09.262815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.261497 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 22 09:19:09.262815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.261713 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 22 09:19:09.262815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.261843 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 22 09:19:09.262815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.261913 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 22 09:19:09.262815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.261918 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 22 09:19:09.262815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.262141 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 22 09:19:09.262815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.262320 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 22 09:19:09.262815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.262391 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f8jr6t3povq0a\"" Jul 22 09:19:09.262815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.262488 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 22 09:19:09.262815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.262594 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 22 09:19:09.262815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.262618 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 22 09:19:09.262815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.262736 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 22 09:19:09.265669 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.265645 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 22 09:19:09.277640 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.277610 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 09:19:09.322499 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322452 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.322616 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322506 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.322616 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.322616 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322579 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.322616 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322606 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.322906 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.322906 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322680 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.322906 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322711 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.322906 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzvj2\" (UniqueName: \"kubernetes.io/projected/0df8b68b-a7ea-42fb-be97-914415067532-kube-api-access-dzvj2\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.322906 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322769 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-config\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.322906 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322799 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.322906 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322828 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.322906 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322860 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.322906 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322884 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-web-config\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.323419 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322913 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.323419 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.322981 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0df8b68b-a7ea-42fb-be97-914415067532-config-out\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.323419 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.323005 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.323419 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.323025 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.323419 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.323062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0df8b68b-a7ea-42fb-be97-914415067532-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.323419 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.323084 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.423576 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.423528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0df8b68b-a7ea-42fb-be97-914415067532-config-out\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.423576 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.423575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.423604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.423642 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0df8b68b-a7ea-42fb-be97-914415067532-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.423667 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.423706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.423739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.423771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.423799 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.423822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.423847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.423896 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.423928 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.423968 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvj2\" (UniqueName: \"kubernetes.io/projected/0df8b68b-a7ea-42fb-be97-914415067532-kube-api-access-dzvj2\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.424000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-config\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.424031 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.424061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424766 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.424096 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424766 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.424123 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-web-config\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.424766 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.424155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.425406 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.425142 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.425515 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.425467 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.432749 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.430193 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0df8b68b-a7ea-42fb-be97-914415067532-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.432749 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.430226 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.432749 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.430662 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.432749 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.430701 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.432749 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.431143 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.432749 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.431539 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0df8b68b-a7ea-42fb-be97-914415067532-config-out\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.432749 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.431602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.432749 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.431736 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.432749 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.431899 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-config\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.432749 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.432430 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.432749 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.432452 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.432749 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.432727 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.433470 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.433073 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.433470 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.433220 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.434604 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.434514 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.435609 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.435569 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-web-config\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.435966 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.435911 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.442345 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.442314 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzvj2\" (UniqueName: \"kubernetes.io/projected/0df8b68b-a7ea-42fb-be97-914415067532-kube-api-access-dzvj2\") pod \"prometheus-k8s-0\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.476451 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.476412 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 09:19:09.570565 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.570476 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:09.603437 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:19:09.603409 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e991e9_0547_42ae_8cf8_f98a7fe4b8e2.slice/crio-e14bc6a3e7787e3ee60debd3963c7c897335403fd0effa3bc02920f8274a4567 WatchSource:0}: Error finding container e14bc6a3e7787e3ee60debd3963c7c897335403fd0effa3bc02920f8274a4567: Status 404 returned error can't find the container with id e14bc6a3e7787e3ee60debd3963c7c897335403fd0effa3bc02920f8274a4567 Jul 22 09:19:09.828084 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.828049 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-66c7958d54-c7w2d"] Jul 22 09:19:09.882235 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.882171 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct"] Jul 22 09:19:09.907906 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:09.907871 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fbb65654c-7q7l5"] Jul 22 09:19:09.915545 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:19:09.915516 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0832a10_9283_40e6_bb8e_24b6dd30f4e7.slice/crio-b86a2f701f8703c9896dec77e16fb810594d95d5de39449c061afc5005e7eb49 WatchSource:0}: Error finding container b86a2f701f8703c9896dec77e16fb810594d95d5de39449c061afc5005e7eb49: Status 404 returned error can't find the container with id b86a2f701f8703c9896dec77e16fb810594d95d5de39449c061afc5005e7eb49 Jul 22 09:19:10.136687 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.136644 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c486685d6-7gzss"] Jul 22 09:19:10.149172 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:19:10.149135 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9edf9893_482b_48a8_9aa0_7baf080b74f1.slice/crio-2ace0bb253d6ecadebc934b89c18aec09d45cfaf5a3fab8bc0902c2d54e5ae60 WatchSource:0}: Error finding container 2ace0bb253d6ecadebc934b89c18aec09d45cfaf5a3fab8bc0902c2d54e5ae60: Status 404 returned error can't find the container with id 2ace0bb253d6ecadebc934b89c18aec09d45cfaf5a3fab8bc0902c2d54e5ae60 Jul 22 09:19:10.163957 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.163848 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 09:19:10.168821 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:19:10.168772 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df8b68b_a7ea_42fb_be97_914415067532.slice/crio-cd97b36bd369fafc63d96e3e5c265126fdebfed52bff108477f38a828684128f WatchSource:0}: Error finding container cd97b36bd369fafc63d96e3e5c265126fdebfed52bff108477f38a828684128f: Status 404 returned error can't find the container with id cd97b36bd369fafc63d96e3e5c265126fdebfed52bff108477f38a828684128f Jul 22 09:19:10.338576 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.338503 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" event={"ID":"f0832a10-9283-40e6-bb8e-24b6dd30f4e7","Type":"ContainerStarted","Data":"b86a2f701f8703c9896dec77e16fb810594d95d5de39449c061afc5005e7eb49"} Jul 22 09:19:10.340217 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.340188 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerStarted","Data":"e14bc6a3e7787e3ee60debd3963c7c897335403fd0effa3bc02920f8274a4567"} Jul 22 09:19:10.342114 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.342087 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-km7cq" event={"ID":"ec80e86d-c05c-439a-ad34-c44e4009f060","Type":"ContainerStarted","Data":"693f9c5ec031e290d3b2638426810914eef4c6af40a90934cbaa850819363786"} Jul 22 09:19:10.342207 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.342166 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:19:10.343728 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.343697 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c486685d6-7gzss" event={"ID":"9edf9893-482b-48a8-9aa0-7baf080b74f1","Type":"ContainerStarted","Data":"e077bba082ec2e7b780ff6224a650ce9bc42f3ab4feb25fcda89bf7563cefd62"} Jul 22 09:19:10.343817 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.343739 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c486685d6-7gzss" event={"ID":"9edf9893-482b-48a8-9aa0-7baf080b74f1","Type":"ContainerStarted","Data":"2ace0bb253d6ecadebc934b89c18aec09d45cfaf5a3fab8bc0902c2d54e5ae60"} Jul 22 09:19:10.345756 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.345734 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" event={"ID":"3dae0afa-7c17-4b64-8f0f-fbbf46c632da","Type":"ContainerStarted","Data":"9b507f91771aefd18f65affd04f9993380a0c0b5928571f23f96f6f2137191b3"} Jul 22 09:19:10.347541 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.347469 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd5cff79c-tnkmm" event={"ID":"7de0edbf-64ae-41fa-9fce-ca32ab28a277","Type":"ContainerStarted","Data":"7bd1df7821abc02a0dda7874dd179c8d2f0fd88413b12f674b8790bb7b0f2337"} Jul 22 09:19:10.349844 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.349793 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" event={"ID":"2ad4272f-1391-4472-a944-c3bc4f8ddfb5","Type":"ContainerStarted","Data":"037f713cd2b5e59fdd72265e13b0ae73fd526a22b0fc8b046aeb880ca067d60b"} Jul 22 09:19:10.349965 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.349846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" event={"ID":"2ad4272f-1391-4472-a944-c3bc4f8ddfb5","Type":"ContainerStarted","Data":"ea4d4685565275707f62739dc805efa47eced55daa841a5cccaa51a6ebecff60"} Jul 22 09:19:10.349965 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.349875 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" event={"ID":"2ad4272f-1391-4472-a944-c3bc4f8ddfb5","Type":"ContainerStarted","Data":"46adb25ee44dfce3e19ba0bc11db4bfc2d427f2f28c1fe78fa11fa88cb9cdb23"} Jul 22 09:19:10.352450 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.352295 2564 generic.go:356] "Generic (PLEG): container finished" podID="9730e0f1-91ed-44fa-ae9f-df4e8ed99a15" containerID="6423a1e492f6d5140769430ce90f02ac3a1bf82866f61885f006ce4a19937562" exitCode=0 Jul 22 09:19:10.352450 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.352376 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fc79w" event={"ID":"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15","Type":"ContainerDied","Data":"6423a1e492f6d5140769430ce90f02ac3a1bf82866f61885f006ce4a19937562"} Jul 22 09:19:10.354108 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.354076 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5zw6v" event={"ID":"21a0247b-c2c6-4f3e-a619-2eaaf17a2a06","Type":"ContainerStarted","Data":"01783d30867754dbfa215568f44a0328b8605fbc06f006e0384513a625dc5dc9"} Jul 22 09:19:10.356049 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.356024 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct" event={"ID":"b093f3a3-f6e3-4692-be96-8319e49aea36","Type":"ContainerStarted","Data":"cc07f0dadcd4f4d657815771205ad31b86dc0d77aefb19f3ecfcfa3bc4b44780"} Jul 22 09:19:10.359973 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.359529 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-998d7989b-d5w8r" event={"ID":"86c6c23c-8c3a-42c3-bd8d-33019807a0ed","Type":"ContainerStarted","Data":"ab51df6d606433a74b04a1ea9528e764cb6a9de342e1925c3a978e4706d9eca2"} Jul 22 09:19:10.361598 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.360838 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-km7cq" podStartSLOduration=67.632427981 podStartE2EDuration="1m15.360822397s" totalImagesPullingTime="7.728394416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:10.359861462 +0000 UTC m=+74.857368898" watchObservedRunningTime="2026-07-22 09:19:10.360822397 +0000 UTC m=+74.858329834" Jul 22 09:19:10.363711 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.363683 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerStarted","Data":"cd97b36bd369fafc63d96e3e5c265126fdebfed52bff108477f38a828684128f"} Jul 22 09:19:10.370471 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.370444 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" event={"ID":"21d042ab-2267-4459-9215-01e50bf7eee8","Type":"ContainerStarted","Data":"03eaebf8dd2317cbae4c645c5a289b75b1082b7cedf5f12b7fc815efc1584e0d"} Jul 22 09:19:10.384093 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.383900 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-xbttw" podStartSLOduration=2.807019994 podStartE2EDuration="8.383880115s" totalImagesPullingTime="5.576860121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:19:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:10.381540165 +0000 UTC m=+74.879047602" watchObservedRunningTime="2026-07-22 09:19:10.383880115 +0000 UTC m=+74.881387549" Jul 22 09:19:10.438643 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.438531 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5zw6v" podStartSLOduration=67.003633401 podStartE2EDuration="1m14.4385115s" totalImagesPullingTime="7.434878099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 09:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:10.416452854 +0000 UTC m=+74.913960290" watchObservedRunningTime="2026-07-22 09:19:10.4385115 +0000 UTC m=+74.936018919" Jul 22 09:19:10.442133 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.439396 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bd5cff79c-tnkmm" podStartSLOduration=3.415555487 podStartE2EDuration="13.439383457s" totalImagesPullingTime="10.02382797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:10.439043168 +0000 UTC m=+74.936550606" watchObservedRunningTime="2026-07-22 09:19:10.439383457 +0000 UTC m=+74.936890894" Jul 22 09:19:10.464919 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.464858 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c486685d6-7gzss" podStartSLOduration=2.464827266 podStartE2EDuration="2.464827266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:10.462451075 +0000 UTC m=+74.959958514" watchObservedRunningTime="2026-07-22 09:19:10.464827266 +0000 UTC m=+74.962334703" Jul 22 09:19:10.508183 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.506905 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-998d7989b-d5w8r" podStartSLOduration=2.095568437 podStartE2EDuration="7.50688245s" totalImagesPullingTime="5.411314013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:10.505335197 +0000 UTC m=+75.002842634" watchObservedRunningTime="2026-07-22 09:19:10.50688245 +0000 UTC m=+75.004389887" Jul 22 09:19:10.508183 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:10.507108 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-rbpds" podStartSLOduration=2.781764719 podStartE2EDuration="8.507097912s" totalImagesPullingTime="5.725333193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:19:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:10.48187264 +0000 UTC m=+74.979380076" watchObservedRunningTime="2026-07-22 09:19:10.507097912 +0000 UTC m=+75.004605386" Jul 22 09:19:11.380485 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:11.380440 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerStarted","Data":"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d"} Jul 22 09:19:11.385373 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:11.384774 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerStarted","Data":"863d81082269b1f020fdabc3880b1b4c5d6f01e694c35508c1c868ba96a493f0"} Jul 22 09:19:11.389046 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:11.388861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fc79w" event={"ID":"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15","Type":"ContainerStarted","Data":"6b125153b3b5d639cee22d0ebd20785fab50ea909bf36273bb4d2a1729c2dd08"} Jul 22 09:19:11.389046 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:11.388909 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fc79w" event={"ID":"9730e0f1-91ed-44fa-ae9f-df4e8ed99a15","Type":"ContainerStarted","Data":"840645133a25bf4a97ad3487711cb8f49365017aadce48c31e363523370b0286"} Jul 22 09:19:11.436806 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:11.436728 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fc79w" podStartSLOduration=2.940863917 podStartE2EDuration="9.436707106s" totalImagesPullingTime="6.365422715s" totalInitContainerRuntime="130.420474ms" isStatefulPod=true podCreationTimestamp="2026-07-22 09:19:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:11.436126208 +0000 UTC m=+75.933633643" watchObservedRunningTime="2026-07-22 09:19:11.436707106 +0000 UTC m=+75.934214543" Jul 22 09:19:12.395562 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:12.395464 2564 generic.go:356] "Generic (PLEG): container finished" podID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerID="863d81082269b1f020fdabc3880b1b4c5d6f01e694c35508c1c868ba96a493f0" exitCode=0 Jul 22 09:19:12.396020 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:12.395604 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerDied","Data":"863d81082269b1f020fdabc3880b1b4c5d6f01e694c35508c1c868ba96a493f0"} Jul 22 09:19:12.399142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:12.399069 2564 generic.go:356] "Generic (PLEG): container finished" podID="0df8b68b-a7ea-42fb-be97-914415067532" containerID="fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d" exitCode=0 Jul 22 09:19:12.399212 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:12.399190 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerDied","Data":"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d"} Jul 22 09:19:13.407090 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:13.407046 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" event={"ID":"3dae0afa-7c17-4b64-8f0f-fbbf46c632da","Type":"ContainerStarted","Data":"af30530fc5dab4ce9bcbc19cdeb85ae61d0e3ca249ef4dc2604de49978ccd3e6"} Jul 22 09:19:13.407573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:13.407141 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" event={"ID":"3dae0afa-7c17-4b64-8f0f-fbbf46c632da","Type":"ContainerStarted","Data":"3efd3ddc27332b7522a14638fd7140df5979a57a27730de9b1c104bacb1f8982"} Jul 22 09:19:13.407573 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:13.407183 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" event={"ID":"3dae0afa-7c17-4b64-8f0f-fbbf46c632da","Type":"ContainerStarted","Data":"967263e74fea64d80d405e08bd360e4de989d724eef4adee81f12375fac7e2a0"} Jul 22 09:19:13.409799 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:13.409713 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct" event={"ID":"b093f3a3-f6e3-4692-be96-8319e49aea36","Type":"ContainerStarted","Data":"335cf31f06ce08bd60414e72148c52eaf8b0cccb3ea3e253e88db7340ca14f88"} Jul 22 09:19:13.409799 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:13.409766 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct" Jul 22 09:19:13.411994 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:13.411962 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" event={"ID":"f0832a10-9283-40e6-bb8e-24b6dd30f4e7","Type":"ContainerStarted","Data":"c0ffbcfdeb276657ef12600130ef009ac68c3b973e9385f11ee4ae28a63b3373"} Jul 22 09:19:13.416768 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:13.416745 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct" Jul 22 09:19:13.445464 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:13.445405 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-tqtct" podStartSLOduration=3.6565199120000003 podStartE2EDuration="6.445386521s" totalImagesPullingTime="2.788866609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:13.427402093 +0000 UTC m=+77.924909541" watchObservedRunningTime="2026-07-22 09:19:13.445386521 +0000 UTC m=+77.942893955" Jul 22 09:19:13.463388 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:13.463331 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" podStartSLOduration=3.692063539 podStartE2EDuration="6.463311825s" totalImagesPullingTime="2.771248286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:13.462884616 +0000 UTC m=+77.960392052" watchObservedRunningTime="2026-07-22 09:19:13.463311825 +0000 UTC m=+77.960819262" Jul 22 09:19:14.035076 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:14.034992 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:14.035243 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:14.035176 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:14.043089 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:14.043059 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:14.419685 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:14.419650 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerStarted","Data":"88eb20fade669f7f37a51ec41b0f99382d598f0e11003f4383da419fc87202f9"} Jul 22 09:19:14.424084 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:14.424053 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" event={"ID":"3dae0afa-7c17-4b64-8f0f-fbbf46c632da","Type":"ContainerStarted","Data":"9818632bdd3f688f8cd171d152eb3d54d0824445505d4f8157602f4357c04d3d"} Jul 22 09:19:14.428899 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:14.428874 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:15.433486 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:15.433449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerStarted","Data":"cbe020d848338003c04ccb9f435cdf522c101c2f82874ac881018b7e9fbe1b9a"} Jul 22 09:19:15.433986 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:15.433538 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerStarted","Data":"dc889631c708b9dae0a67bbbd81f8e4a12211f800dc54e6b836dbe94324d66ae"} Jul 22 09:19:15.433986 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:15.433579 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerStarted","Data":"4f5557b8d071ee5bd9f3bb68f8cbad5f4cdc6d419ca8a7feb480d99ed7fbfc6e"} Jul 22 09:19:15.437003 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:15.436968 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" event={"ID":"3dae0afa-7c17-4b64-8f0f-fbbf46c632da","Type":"ContainerStarted","Data":"3f0fc6810f5771d6d8d15f1a5bcf9ae7a054f12256cb0d321ba7d095be774f54"} Jul 22 09:19:15.437226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:15.437045 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" event={"ID":"3dae0afa-7c17-4b64-8f0f-fbbf46c632da","Type":"ContainerStarted","Data":"59167d5c066c8e3da909145320df070acab353f74cbf33a30f25acd5d9e3373d"} Jul 22 09:19:15.437365 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:15.437346 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:15.463004 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:15.462919 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" podStartSLOduration=5.26001955 podStartE2EDuration="9.462897655s" totalImagesPullingTime="4.202878105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:19:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:15.460126594 +0000 UTC m=+79.957634042" watchObservedRunningTime="2026-07-22 09:19:15.462897655 +0000 UTC m=+79.960405093" Jul 22 09:19:16.444335 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:16.444298 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerStarted","Data":"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc"} Jul 22 09:19:16.444721 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:16.444386 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerStarted","Data":"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5"} Jul 22 09:19:16.444721 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:16.444435 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerStarted","Data":"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477"} Jul 22 09:19:16.444721 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:16.444473 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerStarted","Data":"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4"} Jul 22 09:19:16.444721 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:16.444511 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerStarted","Data":"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766"} Jul 22 09:19:16.444721 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:16.444548 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerStarted","Data":"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4"} Jul 22 09:19:16.447517 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:16.447489 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerStarted","Data":"31bcb88dd3c8d48f538560dffbcda1579073923f775ce3eb94c63195a88ff76c"} Jul 22 09:19:16.447623 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:16.447556 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerStarted","Data":"6451214bef77b1b8bef2c7c035947d3a05398bb2038842558ed86d942e3908a6"} Jul 22 09:19:16.481817 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:16.481766 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.187520145 podStartE2EDuration="7.481747445s" totalImagesPullingTime="4.271000309s" totalInitContainerRuntime="23.226991ms" isStatefulPod=false podCreationTimestamp="2026-07-22 09:19:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:16.478761406 +0000 UTC m=+80.976268842" watchObservedRunningTime="2026-07-22 09:19:16.481747445 +0000 UTC m=+80.979254881" Jul 22 09:19:16.507843 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:16.507783 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=9.068192052 podStartE2EDuration="12.507765704s" totalImagesPullingTime="3.420863345s" totalInitContainerRuntime="18.710307ms" isStatefulPod=false podCreationTimestamp="2026-07-22 09:19:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:19:16.506726325 +0000 UTC m=+81.004233762" watchObservedRunningTime="2026-07-22 09:19:16.507765704 +0000 UTC m=+81.005273144" Jul 22 09:19:18.219337 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:18.219214 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:19:19.259385 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:19.259338 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:19.259844 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:19.259406 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:19.264140 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:19.264121 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:19.466709 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:19.466684 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:19:19.519778 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:19.519700 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-998d7989b-d5w8r"] Jul 22 09:19:19.571152 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:19.571120 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:19:21.453060 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:21.453030 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-66c7958d54-c7w2d" Jul 22 09:19:27.585132 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:27.585098 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:27.585528 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:27.585167 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:35.378752 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:35.378681 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bd5cff79c-tnkmm" podUID="7de0edbf-64ae-41fa-9fce-ca32ab28a277" containerName="console" containerID="cri-o://7bd1df7821abc02a0dda7874dd179c8d2f0fd88413b12f674b8790bb7b0f2337" gracePeriod=15 Jul 22 09:19:36.169810 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.169788 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:19:36.198585 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.198559 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-service-ca\") pod \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " Jul 22 09:19:36.198735 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.198601 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-oauth-serving-cert\") pod \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " Jul 22 09:19:36.198735 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.198717 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-serving-cert\") pod \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " Jul 22 09:19:36.198865 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.198848 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-config\") pod \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " Jul 22 09:19:36.198925 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.198910 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn5ht\" (UniqueName: \"kubernetes.io/projected/7de0edbf-64ae-41fa-9fce-ca32ab28a277-kube-api-access-zn5ht\") pod \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " Jul 22 09:19:36.198995 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.198962 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7de0edbf-64ae-41fa-9fce-ca32ab28a277" (UID: "7de0edbf-64ae-41fa-9fce-ca32ab28a277"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:19:36.198995 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.198974 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-service-ca" (OuterVolumeSpecName: "service-ca") pod "7de0edbf-64ae-41fa-9fce-ca32ab28a277" (UID: "7de0edbf-64ae-41fa-9fce-ca32ab28a277"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:19:36.198995 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.198989 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-oauth-config\") pod \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\" (UID: \"7de0edbf-64ae-41fa-9fce-ca32ab28a277\") " Jul 22 09:19:36.199257 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.199220 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-config" (OuterVolumeSpecName: "console-config") pod "7de0edbf-64ae-41fa-9fce-ca32ab28a277" (UID: "7de0edbf-64ae-41fa-9fce-ca32ab28a277"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:19:36.199376 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.199299 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-service-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:19:36.199376 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.199320 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-oauth-serving-cert\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:19:36.201012 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.200986 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7de0edbf-64ae-41fa-9fce-ca32ab28a277" (UID: "7de0edbf-64ae-41fa-9fce-ca32ab28a277"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:19:36.201113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.201095 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7de0edbf-64ae-41fa-9fce-ca32ab28a277" (UID: "7de0edbf-64ae-41fa-9fce-ca32ab28a277"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:19:36.201163 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.201143 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de0edbf-64ae-41fa-9fce-ca32ab28a277-kube-api-access-zn5ht" (OuterVolumeSpecName: "kube-api-access-zn5ht") pod "7de0edbf-64ae-41fa-9fce-ca32ab28a277" (UID: "7de0edbf-64ae-41fa-9fce-ca32ab28a277"). InnerVolumeSpecName "kube-api-access-zn5ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:19:36.300279 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.300243 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-config\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:19:36.300279 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.300273 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn5ht\" (UniqueName: \"kubernetes.io/projected/7de0edbf-64ae-41fa-9fce-ca32ab28a277-kube-api-access-zn5ht\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:19:36.300279 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.300283 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-oauth-config\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:19:36.300279 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.300291 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7de0edbf-64ae-41fa-9fce-ca32ab28a277-console-serving-cert\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:19:36.532790 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.532753 2564 generic.go:356] "Generic (PLEG): container finished" podID="7de0edbf-64ae-41fa-9fce-ca32ab28a277" containerID="7bd1df7821abc02a0dda7874dd179c8d2f0fd88413b12f674b8790bb7b0f2337" exitCode=0 Jul 22 09:19:36.533194 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.532801 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd5cff79c-tnkmm" Jul 22 09:19:36.533194 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.533129 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd5cff79c-tnkmm" event={"ID":"7de0edbf-64ae-41fa-9fce-ca32ab28a277","Type":"ContainerDied","Data":"7bd1df7821abc02a0dda7874dd179c8d2f0fd88413b12f674b8790bb7b0f2337"} Jul 22 09:19:36.533326 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.533202 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd5cff79c-tnkmm" event={"ID":"7de0edbf-64ae-41fa-9fce-ca32ab28a277","Type":"ContainerDied","Data":"3269bc9bd0a9f72570d0717f558cab89d1e246573a1842b334fc751f03c856e7"} Jul 22 09:19:36.533326 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.533250 2564 scope.go:117] "RemoveContainer" containerID="7bd1df7821abc02a0dda7874dd179c8d2f0fd88413b12f674b8790bb7b0f2337" Jul 22 09:19:36.549101 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.549071 2564 scope.go:117] "RemoveContainer" containerID="7bd1df7821abc02a0dda7874dd179c8d2f0fd88413b12f674b8790bb7b0f2337" Jul 22 09:19:36.549395 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:19:36.549370 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd1df7821abc02a0dda7874dd179c8d2f0fd88413b12f674b8790bb7b0f2337\": container with ID starting with 7bd1df7821abc02a0dda7874dd179c8d2f0fd88413b12f674b8790bb7b0f2337 not found: ID does not exist" containerID="7bd1df7821abc02a0dda7874dd179c8d2f0fd88413b12f674b8790bb7b0f2337" Jul 22 09:19:36.549495 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.549399 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd1df7821abc02a0dda7874dd179c8d2f0fd88413b12f674b8790bb7b0f2337"} err="failed to get container status \"7bd1df7821abc02a0dda7874dd179c8d2f0fd88413b12f674b8790bb7b0f2337\": rpc error: code = NotFound desc = could not find container \"7bd1df7821abc02a0dda7874dd179c8d2f0fd88413b12f674b8790bb7b0f2337\": container with ID starting with 7bd1df7821abc02a0dda7874dd179c8d2f0fd88413b12f674b8790bb7b0f2337 not found: ID does not exist" Jul 22 09:19:36.560480 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.560446 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bd5cff79c-tnkmm"] Jul 22 09:19:36.564615 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:36.564592 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bd5cff79c-tnkmm"] Jul 22 09:19:38.013856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:38.013823 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7de0edbf-64ae-41fa-9fce-ca32ab28a277" path="/var/lib/kubelet/pods/7de0edbf-64ae-41fa-9fce-ca32ab28a277/volumes" Jul 22 09:19:41.391985 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:41.391954 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-km7cq" Jul 22 09:19:44.541088 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.541042 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-998d7989b-d5w8r" podUID="86c6c23c-8c3a-42c3-bd8d-33019807a0ed" containerName="console" containerID="cri-o://ab51df6d606433a74b04a1ea9528e764cb6a9de342e1925c3a978e4706d9eca2" gracePeriod=15 Jul 22 09:19:44.780161 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.780136 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:44.875344 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.875265 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-oauth-config\") pod \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " Jul 22 09:19:44.875344 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.875330 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-service-ca\") pod \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " Jul 22 09:19:44.875551 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.875382 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-oauth-serving-cert\") pod \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " Jul 22 09:19:44.875551 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.875409 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-serving-cert\") pod \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " Jul 22 09:19:44.875551 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.875443 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9l4f\" (UniqueName: \"kubernetes.io/projected/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-kube-api-access-w9l4f\") pod \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " Jul 22 09:19:44.875551 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.875484 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-config\") pod \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " Jul 22 09:19:44.875551 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.875520 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-trusted-ca-bundle\") pod \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\" (UID: \"86c6c23c-8c3a-42c3-bd8d-33019807a0ed\") " Jul 22 09:19:44.875789 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.875762 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-service-ca" (OuterVolumeSpecName: "service-ca") pod "86c6c23c-8c3a-42c3-bd8d-33019807a0ed" (UID: "86c6c23c-8c3a-42c3-bd8d-33019807a0ed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:19:44.875836 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.875770 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "86c6c23c-8c3a-42c3-bd8d-33019807a0ed" (UID: "86c6c23c-8c3a-42c3-bd8d-33019807a0ed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:19:44.876103 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.876071 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "86c6c23c-8c3a-42c3-bd8d-33019807a0ed" (UID: "86c6c23c-8c3a-42c3-bd8d-33019807a0ed"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:19:44.876202 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.876180 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-config" (OuterVolumeSpecName: "console-config") pod "86c6c23c-8c3a-42c3-bd8d-33019807a0ed" (UID: "86c6c23c-8c3a-42c3-bd8d-33019807a0ed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:19:44.877471 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.877448 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "86c6c23c-8c3a-42c3-bd8d-33019807a0ed" (UID: "86c6c23c-8c3a-42c3-bd8d-33019807a0ed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:19:44.877574 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.877518 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-kube-api-access-w9l4f" (OuterVolumeSpecName: "kube-api-access-w9l4f") pod "86c6c23c-8c3a-42c3-bd8d-33019807a0ed" (UID: "86c6c23c-8c3a-42c3-bd8d-33019807a0ed"). InnerVolumeSpecName "kube-api-access-w9l4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:19:44.877574 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.877535 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "86c6c23c-8c3a-42c3-bd8d-33019807a0ed" (UID: "86c6c23c-8c3a-42c3-bd8d-33019807a0ed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:19:44.976197 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.976158 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-oauth-config\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:19:44.976197 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.976192 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-service-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:19:44.976197 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.976206 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-oauth-serving-cert\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:19:44.976408 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.976218 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-serving-cert\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:19:44.976408 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.976229 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9l4f\" (UniqueName: \"kubernetes.io/projected/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-kube-api-access-w9l4f\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:19:44.976408 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.976241 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-console-config\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:19:44.976408 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:44.976252 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86c6c23c-8c3a-42c3-bd8d-33019807a0ed-trusted-ca-bundle\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:19:45.573147 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:45.573108 2564 generic.go:356] "Generic (PLEG): container finished" podID="86c6c23c-8c3a-42c3-bd8d-33019807a0ed" containerID="ab51df6d606433a74b04a1ea9528e764cb6a9de342e1925c3a978e4706d9eca2" exitCode=0 Jul 22 09:19:45.573605 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:45.573159 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-998d7989b-d5w8r" Jul 22 09:19:45.573605 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:45.573197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-998d7989b-d5w8r" event={"ID":"86c6c23c-8c3a-42c3-bd8d-33019807a0ed","Type":"ContainerDied","Data":"ab51df6d606433a74b04a1ea9528e764cb6a9de342e1925c3a978e4706d9eca2"} Jul 22 09:19:45.573605 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:45.573247 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-998d7989b-d5w8r" event={"ID":"86c6c23c-8c3a-42c3-bd8d-33019807a0ed","Type":"ContainerDied","Data":"5a9e532f188adab1a80f46562d6fd3b8b5778ff991ca3f96dc4c822715d86064"} Jul 22 09:19:45.573605 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:45.573267 2564 scope.go:117] "RemoveContainer" containerID="ab51df6d606433a74b04a1ea9528e764cb6a9de342e1925c3a978e4706d9eca2" Jul 22 09:19:45.581724 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:45.581708 2564 scope.go:117] "RemoveContainer" containerID="ab51df6d606433a74b04a1ea9528e764cb6a9de342e1925c3a978e4706d9eca2" Jul 22 09:19:45.581990 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:19:45.581969 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab51df6d606433a74b04a1ea9528e764cb6a9de342e1925c3a978e4706d9eca2\": container with ID starting with ab51df6d606433a74b04a1ea9528e764cb6a9de342e1925c3a978e4706d9eca2 not found: ID does not exist" containerID="ab51df6d606433a74b04a1ea9528e764cb6a9de342e1925c3a978e4706d9eca2" Jul 22 09:19:45.582039 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:45.582001 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab51df6d606433a74b04a1ea9528e764cb6a9de342e1925c3a978e4706d9eca2"} err="failed to get container status \"ab51df6d606433a74b04a1ea9528e764cb6a9de342e1925c3a978e4706d9eca2\": rpc error: code = NotFound desc = could not find container \"ab51df6d606433a74b04a1ea9528e764cb6a9de342e1925c3a978e4706d9eca2\": container with ID starting with ab51df6d606433a74b04a1ea9528e764cb6a9de342e1925c3a978e4706d9eca2 not found: ID does not exist" Jul 22 09:19:45.595123 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:45.595098 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-998d7989b-d5w8r"] Jul 22 09:19:45.599383 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:45.599363 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-998d7989b-d5w8r"] Jul 22 09:19:46.014288 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:46.014258 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86c6c23c-8c3a-42c3-bd8d-33019807a0ed" path="/var/lib/kubelet/pods/86c6c23c-8c3a-42c3-bd8d-33019807a0ed/volumes" Jul 22 09:19:47.590059 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:47.590028 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:19:47.593955 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:19:47.593920 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7fbb65654c-7q7l5" Jul 22 09:20:09.570790 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:09.570752 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:09.589671 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:09.589637 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:09.687484 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:09.687460 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:23.354753 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.354719 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 09:20:23.355270 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.355059 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="alertmanager" containerID="cri-o://88eb20fade669f7f37a51ec41b0f99382d598f0e11003f4383da419fc87202f9" gracePeriod=120 Jul 22 09:20:23.355270 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.355109 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="kube-rbac-proxy-metric" containerID="cri-o://6451214bef77b1b8bef2c7c035947d3a05398bb2038842558ed86d942e3908a6" gracePeriod=120 Jul 22 09:20:23.355270 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.355163 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="config-reloader" containerID="cri-o://4f5557b8d071ee5bd9f3bb68f8cbad5f4cdc6d419ca8a7feb480d99ed7fbfc6e" gracePeriod=120 Jul 22 09:20:23.355270 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.355165 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="prom-label-proxy" containerID="cri-o://31bcb88dd3c8d48f538560dffbcda1579073923f775ce3eb94c63195a88ff76c" gracePeriod=120 Jul 22 09:20:23.355270 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.355170 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="kube-rbac-proxy" containerID="cri-o://cbe020d848338003c04ccb9f435cdf522c101c2f82874ac881018b7e9fbe1b9a" gracePeriod=120 Jul 22 09:20:23.355270 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.355135 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="kube-rbac-proxy-web" containerID="cri-o://dc889631c708b9dae0a67bbbd81f8e4a12211f800dc54e6b836dbe94324d66ae" gracePeriod=120 Jul 22 09:20:23.731707 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.731675 2564 generic.go:356] "Generic (PLEG): container finished" podID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerID="31bcb88dd3c8d48f538560dffbcda1579073923f775ce3eb94c63195a88ff76c" exitCode=0 Jul 22 09:20:23.731707 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.731700 2564 generic.go:356] "Generic (PLEG): container finished" podID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerID="6451214bef77b1b8bef2c7c035947d3a05398bb2038842558ed86d942e3908a6" exitCode=0 Jul 22 09:20:23.731707 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.731707 2564 generic.go:356] "Generic (PLEG): container finished" podID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerID="cbe020d848338003c04ccb9f435cdf522c101c2f82874ac881018b7e9fbe1b9a" exitCode=0 Jul 22 09:20:23.731707 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.731714 2564 generic.go:356] "Generic (PLEG): container finished" podID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerID="4f5557b8d071ee5bd9f3bb68f8cbad5f4cdc6d419ca8a7feb480d99ed7fbfc6e" exitCode=0 Jul 22 09:20:23.732004 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.731719 2564 generic.go:356] "Generic (PLEG): container finished" podID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerID="88eb20fade669f7f37a51ec41b0f99382d598f0e11003f4383da419fc87202f9" exitCode=0 Jul 22 09:20:23.732004 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.731751 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerDied","Data":"31bcb88dd3c8d48f538560dffbcda1579073923f775ce3eb94c63195a88ff76c"} Jul 22 09:20:23.732004 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.731824 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerDied","Data":"6451214bef77b1b8bef2c7c035947d3a05398bb2038842558ed86d942e3908a6"} Jul 22 09:20:23.732004 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.731850 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerDied","Data":"cbe020d848338003c04ccb9f435cdf522c101c2f82874ac881018b7e9fbe1b9a"} Jul 22 09:20:23.732004 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.731876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerDied","Data":"4f5557b8d071ee5bd9f3bb68f8cbad5f4cdc6d419ca8a7feb480d99ed7fbfc6e"} Jul 22 09:20:23.732004 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:23.731900 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerDied","Data":"88eb20fade669f7f37a51ec41b0f99382d598f0e11003f4383da419fc87202f9"} Jul 22 09:20:24.592428 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.592405 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.712281 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.712245 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-alertmanager-main-db\") pod \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " Jul 22 09:20:24.712450 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.712321 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy\") pod \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " Jul 22 09:20:24.712450 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.712343 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-tls-assets\") pod \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " Jul 22 09:20:24.712450 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.712372 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-web-config\") pod \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " Jul 22 09:20:24.712450 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.712388 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-config-volume\") pod \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " Jul 22 09:20:24.712665 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.712530 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-config-out\") pod \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " Jul 22 09:20:24.712665 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.712578 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " Jul 22 09:20:24.712665 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.712591 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" (UID: "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:20:24.712665 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.712607 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp5jd\" (UniqueName: \"kubernetes.io/projected/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-kube-api-access-pp5jd\") pod \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " Jul 22 09:20:24.712665 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.712635 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " Jul 22 09:20:24.712665 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.712658 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-metrics-client-ca\") pod \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " Jul 22 09:20:24.712993 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.712701 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-alertmanager-trusted-ca-bundle\") pod \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " Jul 22 09:20:24.712993 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.712731 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-cluster-tls-config\") pod \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " Jul 22 09:20:24.712993 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.712769 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-main-tls\") pod \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\" (UID: \"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2\") " Jul 22 09:20:24.713145 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.713059 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-alertmanager-main-db\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:24.713726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.713427 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" (UID: "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:20:24.713726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.713457 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" (UID: "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:20:24.715394 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.715367 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-config-volume" (OuterVolumeSpecName: "config-volume") pod "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" (UID: "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:24.715804 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.715679 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" (UID: "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:20:24.715804 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.715776 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" (UID: "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:24.716001 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.715978 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-config-out" (OuterVolumeSpecName: "config-out") pod "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" (UID: "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:20:24.716079 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.716010 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" (UID: "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:24.716155 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.716125 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" (UID: "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:24.717001 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.716964 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" (UID: "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:24.717236 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.717205 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-kube-api-access-pp5jd" (OuterVolumeSpecName: "kube-api-access-pp5jd") pod "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" (UID: "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2"). InnerVolumeSpecName "kube-api-access-pp5jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:20:24.719886 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.719759 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" (UID: "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:24.726239 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.726218 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-web-config" (OuterVolumeSpecName: "web-config") pod "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" (UID: "26e991e9-0547-42ae-8cf8-f98a7fe4b8e2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:24.742792 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.742765 2564 generic.go:356] "Generic (PLEG): container finished" podID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerID="dc889631c708b9dae0a67bbbd81f8e4a12211f800dc54e6b836dbe94324d66ae" exitCode=0 Jul 22 09:20:24.742878 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.742838 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.742926 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.742838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerDied","Data":"dc889631c708b9dae0a67bbbd81f8e4a12211f800dc54e6b836dbe94324d66ae"} Jul 22 09:20:24.743017 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.743005 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"26e991e9-0547-42ae-8cf8-f98a7fe4b8e2","Type":"ContainerDied","Data":"e14bc6a3e7787e3ee60debd3963c7c897335403fd0effa3bc02920f8274a4567"} Jul 22 09:20:24.743052 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.743043 2564 scope.go:117] "RemoveContainer" containerID="31bcb88dd3c8d48f538560dffbcda1579073923f775ce3eb94c63195a88ff76c" Jul 22 09:20:24.751022 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.750951 2564 scope.go:117] "RemoveContainer" containerID="6451214bef77b1b8bef2c7c035947d3a05398bb2038842558ed86d942e3908a6" Jul 22 09:20:24.757479 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.757461 2564 scope.go:117] "RemoveContainer" containerID="cbe020d848338003c04ccb9f435cdf522c101c2f82874ac881018b7e9fbe1b9a" Jul 22 09:20:24.763676 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.763658 2564 scope.go:117] "RemoveContainer" containerID="dc889631c708b9dae0a67bbbd81f8e4a12211f800dc54e6b836dbe94324d66ae" Jul 22 09:20:24.769919 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.769904 2564 scope.go:117] "RemoveContainer" containerID="4f5557b8d071ee5bd9f3bb68f8cbad5f4cdc6d419ca8a7feb480d99ed7fbfc6e" Jul 22 09:20:24.776190 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.776171 2564 scope.go:117] "RemoveContainer" containerID="88eb20fade669f7f37a51ec41b0f99382d598f0e11003f4383da419fc87202f9" Jul 22 09:20:24.782673 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.782658 2564 scope.go:117] "RemoveContainer" containerID="863d81082269b1f020fdabc3880b1b4c5d6f01e694c35508c1c868ba96a493f0" Jul 22 09:20:24.789101 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.789087 2564 scope.go:117] "RemoveContainer" containerID="31bcb88dd3c8d48f538560dffbcda1579073923f775ce3eb94c63195a88ff76c" Jul 22 09:20:24.789390 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:20:24.789373 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31bcb88dd3c8d48f538560dffbcda1579073923f775ce3eb94c63195a88ff76c\": container with ID starting with 31bcb88dd3c8d48f538560dffbcda1579073923f775ce3eb94c63195a88ff76c not found: ID does not exist" containerID="31bcb88dd3c8d48f538560dffbcda1579073923f775ce3eb94c63195a88ff76c" Jul 22 09:20:24.789448 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.789396 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31bcb88dd3c8d48f538560dffbcda1579073923f775ce3eb94c63195a88ff76c"} err="failed to get container status \"31bcb88dd3c8d48f538560dffbcda1579073923f775ce3eb94c63195a88ff76c\": rpc error: code = NotFound desc = could not find container \"31bcb88dd3c8d48f538560dffbcda1579073923f775ce3eb94c63195a88ff76c\": container with ID starting with 31bcb88dd3c8d48f538560dffbcda1579073923f775ce3eb94c63195a88ff76c not found: ID does not exist" Jul 22 09:20:24.789448 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.789412 2564 scope.go:117] "RemoveContainer" containerID="6451214bef77b1b8bef2c7c035947d3a05398bb2038842558ed86d942e3908a6" Jul 22 09:20:24.789637 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:20:24.789623 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6451214bef77b1b8bef2c7c035947d3a05398bb2038842558ed86d942e3908a6\": container with ID starting with 6451214bef77b1b8bef2c7c035947d3a05398bb2038842558ed86d942e3908a6 not found: ID does not exist" containerID="6451214bef77b1b8bef2c7c035947d3a05398bb2038842558ed86d942e3908a6" Jul 22 09:20:24.789684 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.789640 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6451214bef77b1b8bef2c7c035947d3a05398bb2038842558ed86d942e3908a6"} err="failed to get container status \"6451214bef77b1b8bef2c7c035947d3a05398bb2038842558ed86d942e3908a6\": rpc error: code = NotFound desc = could not find container \"6451214bef77b1b8bef2c7c035947d3a05398bb2038842558ed86d942e3908a6\": container with ID starting with 6451214bef77b1b8bef2c7c035947d3a05398bb2038842558ed86d942e3908a6 not found: ID does not exist" Jul 22 09:20:24.789684 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.789654 2564 scope.go:117] "RemoveContainer" containerID="cbe020d848338003c04ccb9f435cdf522c101c2f82874ac881018b7e9fbe1b9a" Jul 22 09:20:24.789856 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:20:24.789840 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe020d848338003c04ccb9f435cdf522c101c2f82874ac881018b7e9fbe1b9a\": container with ID starting with cbe020d848338003c04ccb9f435cdf522c101c2f82874ac881018b7e9fbe1b9a not found: ID does not exist" containerID="cbe020d848338003c04ccb9f435cdf522c101c2f82874ac881018b7e9fbe1b9a" Jul 22 09:20:24.789892 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.789859 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe020d848338003c04ccb9f435cdf522c101c2f82874ac881018b7e9fbe1b9a"} err="failed to get container status \"cbe020d848338003c04ccb9f435cdf522c101c2f82874ac881018b7e9fbe1b9a\": rpc error: code = NotFound desc = could not find container \"cbe020d848338003c04ccb9f435cdf522c101c2f82874ac881018b7e9fbe1b9a\": container with ID starting with cbe020d848338003c04ccb9f435cdf522c101c2f82874ac881018b7e9fbe1b9a not found: ID does not exist" Jul 22 09:20:24.789892 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.789871 2564 scope.go:117] "RemoveContainer" containerID="dc889631c708b9dae0a67bbbd81f8e4a12211f800dc54e6b836dbe94324d66ae" Jul 22 09:20:24.790103 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:20:24.790088 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc889631c708b9dae0a67bbbd81f8e4a12211f800dc54e6b836dbe94324d66ae\": container with ID starting with dc889631c708b9dae0a67bbbd81f8e4a12211f800dc54e6b836dbe94324d66ae not found: ID does not exist" containerID="dc889631c708b9dae0a67bbbd81f8e4a12211f800dc54e6b836dbe94324d66ae" Jul 22 09:20:24.790140 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.790106 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc889631c708b9dae0a67bbbd81f8e4a12211f800dc54e6b836dbe94324d66ae"} err="failed to get container status \"dc889631c708b9dae0a67bbbd81f8e4a12211f800dc54e6b836dbe94324d66ae\": rpc error: code = NotFound desc = could not find container \"dc889631c708b9dae0a67bbbd81f8e4a12211f800dc54e6b836dbe94324d66ae\": container with ID starting with dc889631c708b9dae0a67bbbd81f8e4a12211f800dc54e6b836dbe94324d66ae not found: ID does not exist" Jul 22 09:20:24.790140 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.790117 2564 scope.go:117] "RemoveContainer" containerID="4f5557b8d071ee5bd9f3bb68f8cbad5f4cdc6d419ca8a7feb480d99ed7fbfc6e" Jul 22 09:20:24.790327 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:20:24.790312 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f5557b8d071ee5bd9f3bb68f8cbad5f4cdc6d419ca8a7feb480d99ed7fbfc6e\": container with ID starting with 4f5557b8d071ee5bd9f3bb68f8cbad5f4cdc6d419ca8a7feb480d99ed7fbfc6e not found: ID does not exist" containerID="4f5557b8d071ee5bd9f3bb68f8cbad5f4cdc6d419ca8a7feb480d99ed7fbfc6e" Jul 22 09:20:24.790369 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.790330 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f5557b8d071ee5bd9f3bb68f8cbad5f4cdc6d419ca8a7feb480d99ed7fbfc6e"} err="failed to get container status \"4f5557b8d071ee5bd9f3bb68f8cbad5f4cdc6d419ca8a7feb480d99ed7fbfc6e\": rpc error: code = NotFound desc = could not find container \"4f5557b8d071ee5bd9f3bb68f8cbad5f4cdc6d419ca8a7feb480d99ed7fbfc6e\": container with ID starting with 4f5557b8d071ee5bd9f3bb68f8cbad5f4cdc6d419ca8a7feb480d99ed7fbfc6e not found: ID does not exist" Jul 22 09:20:24.790369 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.790341 2564 scope.go:117] "RemoveContainer" containerID="88eb20fade669f7f37a51ec41b0f99382d598f0e11003f4383da419fc87202f9" Jul 22 09:20:24.790539 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:20:24.790523 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88eb20fade669f7f37a51ec41b0f99382d598f0e11003f4383da419fc87202f9\": container with ID starting with 88eb20fade669f7f37a51ec41b0f99382d598f0e11003f4383da419fc87202f9 not found: ID does not exist" containerID="88eb20fade669f7f37a51ec41b0f99382d598f0e11003f4383da419fc87202f9" Jul 22 09:20:24.790580 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.790543 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88eb20fade669f7f37a51ec41b0f99382d598f0e11003f4383da419fc87202f9"} err="failed to get container status \"88eb20fade669f7f37a51ec41b0f99382d598f0e11003f4383da419fc87202f9\": rpc error: code = NotFound desc = could not find container \"88eb20fade669f7f37a51ec41b0f99382d598f0e11003f4383da419fc87202f9\": container with ID starting with 88eb20fade669f7f37a51ec41b0f99382d598f0e11003f4383da419fc87202f9 not found: ID does not exist" Jul 22 09:20:24.790580 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.790555 2564 scope.go:117] "RemoveContainer" containerID="863d81082269b1f020fdabc3880b1b4c5d6f01e694c35508c1c868ba96a493f0" Jul 22 09:20:24.790749 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:20:24.790734 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863d81082269b1f020fdabc3880b1b4c5d6f01e694c35508c1c868ba96a493f0\": container with ID starting with 863d81082269b1f020fdabc3880b1b4c5d6f01e694c35508c1c868ba96a493f0 not found: ID does not exist" containerID="863d81082269b1f020fdabc3880b1b4c5d6f01e694c35508c1c868ba96a493f0" Jul 22 09:20:24.790786 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.790751 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863d81082269b1f020fdabc3880b1b4c5d6f01e694c35508c1c868ba96a493f0"} err="failed to get container status \"863d81082269b1f020fdabc3880b1b4c5d6f01e694c35508c1c868ba96a493f0\": rpc error: code = NotFound desc = could not find container \"863d81082269b1f020fdabc3880b1b4c5d6f01e694c35508c1c868ba96a493f0\": container with ID starting with 863d81082269b1f020fdabc3880b1b4c5d6f01e694c35508c1c868ba96a493f0 not found: ID does not exist" Jul 22 09:20:24.813780 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.813754 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-main-tls\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:24.813780 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.813775 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:24.813907 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.813785 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-tls-assets\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:24.813907 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.813809 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-web-config\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:24.813907 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.813818 2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-config-volume\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:24.813907 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.813832 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-config-out\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:24.813907 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.813843 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:24.813907 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.813853 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp5jd\" (UniqueName: \"kubernetes.io/projected/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-kube-api-access-pp5jd\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:24.813907 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.813867 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:24.813907 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.813876 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-metrics-client-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:24.813907 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.813885 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:24.813907 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.813899 2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2-cluster-tls-config\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:24.815202 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.815184 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 09:20:24.826472 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.826447 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 09:20:24.901134 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901107 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 09:20:24.901402 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901391 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="config-reloader" Jul 22 09:20:24.901443 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901403 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="config-reloader" Jul 22 09:20:24.901443 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901412 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="prom-label-proxy" Jul 22 09:20:24.901443 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901418 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="prom-label-proxy" Jul 22 09:20:24.901443 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901424 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="init-config-reloader" Jul 22 09:20:24.901443 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901430 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="init-config-reloader" Jul 22 09:20:24.901443 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901442 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="kube-rbac-proxy" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901448 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="kube-rbac-proxy" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901454 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de0edbf-64ae-41fa-9fce-ca32ab28a277" containerName="console" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901459 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de0edbf-64ae-41fa-9fce-ca32ab28a277" containerName="console" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901472 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86c6c23c-8c3a-42c3-bd8d-33019807a0ed" containerName="console" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901477 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="86c6c23c-8c3a-42c3-bd8d-33019807a0ed" containerName="console" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901483 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="alertmanager" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901487 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="alertmanager" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901493 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="kube-rbac-proxy-web" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901499 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="kube-rbac-proxy-web" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901504 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="kube-rbac-proxy-metric" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901509 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="kube-rbac-proxy-metric" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901559 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de0edbf-64ae-41fa-9fce-ca32ab28a277" containerName="console" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901566 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="86c6c23c-8c3a-42c3-bd8d-33019807a0ed" containerName="console" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901572 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="kube-rbac-proxy" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901577 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="prom-label-proxy" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901583 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="kube-rbac-proxy-metric" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901588 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="config-reloader" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901593 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="alertmanager" Jul 22 09:20:24.901650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.901599 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" containerName="kube-rbac-proxy-web" Jul 22 09:20:24.905157 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.905143 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.908781 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.908715 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 09:20:24.908781 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.908733 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 09:20:24.909310 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.909077 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wwzkn\"" Jul 22 09:20:24.909310 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.909110 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 09:20:24.909488 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.909442 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 09:20:24.909683 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.909549 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 09:20:24.910095 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.910079 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 09:20:24.910844 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.910416 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 09:20:24.910844 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.910748 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 09:20:24.915072 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.915049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9519484e-2793-4641-8937-82d24bed53b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.915149 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.915084 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.915149 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.915114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9519484e-2793-4641-8937-82d24bed53b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.915149 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.915139 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9519484e-2793-4641-8937-82d24bed53b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.915266 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.915180 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.915266 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.915194 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.915266 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.915219 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.915266 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.915253 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.915378 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.915342 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlfst\" (UniqueName: \"kubernetes.io/projected/9519484e-2793-4641-8937-82d24bed53b3-kube-api-access-mlfst\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.915378 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.915368 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.915444 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.915428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9519484e-2793-4641-8937-82d24bed53b3-config-out\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.915477 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.915449 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-web-config\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.915477 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.915463 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9519484e-2793-4641-8937-82d24bed53b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:24.915925 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.915904 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 09:20:24.931283 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:24.931262 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 09:20:25.016567 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.016478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9519484e-2793-4641-8937-82d24bed53b3-config-out\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.016567 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.016522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-web-config\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.016778 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.016646 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9519484e-2793-4641-8937-82d24bed53b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.016778 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.016704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9519484e-2793-4641-8937-82d24bed53b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.016778 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.016742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.016958 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.016782 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9519484e-2793-4641-8937-82d24bed53b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.016958 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.016900 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9519484e-2793-4641-8937-82d24bed53b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.017070 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.016975 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.017070 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.017006 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.017070 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.017042 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.017214 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.017069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.017214 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.017097 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9519484e-2793-4641-8937-82d24bed53b3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.017214 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.017100 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlfst\" (UniqueName: \"kubernetes.io/projected/9519484e-2793-4641-8937-82d24bed53b3-kube-api-access-mlfst\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.017214 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.017160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.017503 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.017482 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9519484e-2793-4641-8937-82d24bed53b3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.017635 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.017613 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9519484e-2793-4641-8937-82d24bed53b3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.019392 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.019365 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9519484e-2793-4641-8937-82d24bed53b3-config-out\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.019634 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.019610 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-web-config\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.019728 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.019701 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.019914 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.019888 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9519484e-2793-4641-8937-82d24bed53b3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.020028 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.019984 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.020285 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.020264 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.020388 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.020370 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-config-volume\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.020420 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.020375 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.021344 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.021324 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9519484e-2793-4641-8937-82d24bed53b3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.028306 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.028285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlfst\" (UniqueName: \"kubernetes.io/projected/9519484e-2793-4641-8937-82d24bed53b3-kube-api-access-mlfst\") pod \"alertmanager-main-0\" (UID: \"9519484e-2793-4641-8937-82d24bed53b3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.215704 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.215673 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 09:20:25.355172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.355145 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 09:20:25.357086 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:20:25.357061 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9519484e_2793_4641_8937_82d24bed53b3.slice/crio-a56cdcda7f114382047914cb88991341ad9603d6f67a0c69785d5afecc44c410 WatchSource:0}: Error finding container a56cdcda7f114382047914cb88991341ad9603d6f67a0c69785d5afecc44c410: Status 404 returned error can't find the container with id a56cdcda7f114382047914cb88991341ad9603d6f67a0c69785d5afecc44c410 Jul 22 09:20:25.747567 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.747534 2564 generic.go:356] "Generic (PLEG): container finished" podID="9519484e-2793-4641-8937-82d24bed53b3" containerID="dfa957a5db30068e8aea6fed2bd6417c81444de68df0498822f4b819c851b8c8" exitCode=0 Jul 22 09:20:25.747996 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.747625 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9519484e-2793-4641-8937-82d24bed53b3","Type":"ContainerDied","Data":"dfa957a5db30068e8aea6fed2bd6417c81444de68df0498822f4b819c851b8c8"} Jul 22 09:20:25.747996 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:25.747697 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9519484e-2793-4641-8937-82d24bed53b3","Type":"ContainerStarted","Data":"a56cdcda7f114382047914cb88991341ad9603d6f67a0c69785d5afecc44c410"} Jul 22 09:20:26.013771 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:26.013742 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26e991e9-0547-42ae-8cf8-f98a7fe4b8e2" path="/var/lib/kubelet/pods/26e991e9-0547-42ae-8cf8-f98a7fe4b8e2/volumes" Jul 22 09:20:26.755284 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:26.755250 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9519484e-2793-4641-8937-82d24bed53b3","Type":"ContainerStarted","Data":"3b317501c7fa3f0723b62654152a15dc2350a4d7db3dc4e01098cd510590543a"} Jul 22 09:20:26.755654 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:26.755316 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9519484e-2793-4641-8937-82d24bed53b3","Type":"ContainerStarted","Data":"5cd31fc9e0b3758683196498fe83e1aa980f9bfa0dc57a8c85054c5b50f9aadd"} Jul 22 09:20:26.755654 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:26.755356 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9519484e-2793-4641-8937-82d24bed53b3","Type":"ContainerStarted","Data":"8079cf3d1de35737ff796bc4233993ef9ca79425a0cf2da989c17bcf6e411738"} Jul 22 09:20:26.755654 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:26.755384 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9519484e-2793-4641-8937-82d24bed53b3","Type":"ContainerStarted","Data":"2738ef33af2c84fde747140d4a4aac9d75226433cb805392edaafb2d3763c640"} Jul 22 09:20:26.755654 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:26.755410 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9519484e-2793-4641-8937-82d24bed53b3","Type":"ContainerStarted","Data":"ffb4659cf8f147ac45c37c92c697c1d84370c234169bcb63fcc5c6c1b5696aa1"} Jul 22 09:20:26.755654 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:26.755449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9519484e-2793-4641-8937-82d24bed53b3","Type":"ContainerStarted","Data":"41d6dd9ccfa85cc8430aa53ab984d7755b4d8b68a83f18f888497a2360a56ce9"} Jul 22 09:20:26.795759 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:26.795695 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.780059709 podStartE2EDuration="2.795677958s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.618249ms" isStatefulPod=false podCreationTimestamp="2026-07-22 09:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:20:26.794468859 +0000 UTC m=+151.291976495" watchObservedRunningTime="2026-07-22 09:20:26.795677958 +0000 UTC m=+151.293185395" Jul 22 09:20:27.482900 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.482866 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-59b5558bb7-dvc2r"] Jul 22 09:20:27.485224 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.485209 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.492612 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.492583 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 22 09:20:27.492746 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.492611 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wgn7r\"" Jul 22 09:20:27.492746 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.492654 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 22 09:20:27.492746 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.492614 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 22 09:20:27.492746 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.492706 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 22 09:20:27.493664 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.493645 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 22 09:20:27.498885 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.498863 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 22 09:20:27.508705 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.508684 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-59b5558bb7-dvc2r"] Jul 22 09:20:27.538825 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.538799 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.538980 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.538829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-metrics-client-ca\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.538980 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.538855 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-serving-certs-ca-bundle\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.538980 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.538876 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.538980 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.538966 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-secret-telemeter-client\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.539187 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.539038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4khj\" (UniqueName: \"kubernetes.io/projected/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-kube-api-access-b4khj\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.539187 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.539094 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-telemeter-client-tls\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.539187 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.539154 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-federate-client-tls\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.639842 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.639801 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4khj\" (UniqueName: \"kubernetes.io/projected/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-kube-api-access-b4khj\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.640064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.639855 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-telemeter-client-tls\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.640064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.639900 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-federate-client-tls\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.640064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.639953 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.640064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.639977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-metrics-client-ca\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.640064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.640016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-serving-certs-ca-bundle\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.640064 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.640047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.640376 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.640076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-secret-telemeter-client\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.640909 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.640879 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-metrics-client-ca\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.641166 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.641141 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-serving-certs-ca-bundle\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.641921 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.641900 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.643869 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.643842 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-secret-telemeter-client\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.643961 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.643917 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.644367 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.644342 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-federate-client-tls\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.644443 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.644384 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-telemeter-client-tls\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.649635 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.649607 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4khj\" (UniqueName: \"kubernetes.io/projected/628bdc9e-cc5a-41e9-be17-520d10f0a4f6-kube-api-access-b4khj\") pod \"telemeter-client-59b5558bb7-dvc2r\" (UID: \"628bdc9e-cc5a-41e9-be17-520d10f0a4f6\") " pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.766485 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.766407 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 09:20:27.766889 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.766661 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="prometheus" containerID="cri-o://0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4" gracePeriod=600 Jul 22 09:20:27.766889 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.766692 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="kube-rbac-proxy" containerID="cri-o://d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5" gracePeriod=600 Jul 22 09:20:27.766889 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.766729 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="config-reloader" containerID="cri-o://1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766" gracePeriod=600 Jul 22 09:20:27.766889 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.766749 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="kube-rbac-proxy-thanos" containerID="cri-o://23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc" gracePeriod=600 Jul 22 09:20:27.766889 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.766692 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="thanos-sidecar" containerID="cri-o://791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4" gracePeriod=600 Jul 22 09:20:27.766889 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.766737 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="kube-rbac-proxy-web" containerID="cri-o://b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477" gracePeriod=600 Jul 22 09:20:27.794429 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.794214 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" Jul 22 09:20:27.956063 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:27.955967 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-59b5558bb7-dvc2r"] Jul 22 09:20:27.958555 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:20:27.958525 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628bdc9e_cc5a_41e9_be17_520d10f0a4f6.slice/crio-c7071dd2fa2e83570a3f2d00800977676fce5c17ea71412f5b76940e9f0b5930 WatchSource:0}: Error finding container c7071dd2fa2e83570a3f2d00800977676fce5c17ea71412f5b76940e9f0b5930: Status 404 returned error can't find the container with id c7071dd2fa2e83570a3f2d00800977676fce5c17ea71412f5b76940e9f0b5930 Jul 22 09:20:28.020976 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.020892 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.045361 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.045319 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzvj2\" (UniqueName: \"kubernetes.io/projected/0df8b68b-a7ea-42fb-be97-914415067532-kube-api-access-dzvj2\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.045510 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.045370 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-kube-rbac-proxy\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.045510 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.045398 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-thanos-prometheus-http-client-file\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.045763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.045728 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-web-config\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.045876 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.045805 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-grpc-tls\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.045876 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.045848 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-metrics-client-ca\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.046024 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.045887 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-kubelet-serving-ca-bundle\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.046024 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.045914 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-db\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.046024 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.045985 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-1\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.046024 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.046009 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0df8b68b-a7ea-42fb-be97-914415067532-config-out\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.046220 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.046064 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-trusted-ca-bundle\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.046220 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.046087 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-config\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.046220 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.046132 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-0\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.046220 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.046159 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.046220 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.046193 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0df8b68b-a7ea-42fb-be97-914415067532-tls-assets\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.046460 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.046221 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-metrics-client-certs\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.046460 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.046254 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.046460 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.046280 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-serving-certs-ca-bundle\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.046460 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.046305 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-tls\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.046460 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.046329 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-2\") pod \"0df8b68b-a7ea-42fb-be97-914415067532\" (UID: \"0df8b68b-a7ea-42fb-be97-914415067532\") " Jul 22 09:20:28.047749 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.047716 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:20:28.047900 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.047872 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:20:28.049539 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.049410 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-config" (OuterVolumeSpecName: "config") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:28.049640 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.049576 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:20:28.049640 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.049487 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:20:28.049795 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.049645 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:20:28.049795 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.049757 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:28.050196 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.050167 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df8b68b-a7ea-42fb-be97-914415067532-kube-api-access-dzvj2" (OuterVolumeSpecName: "kube-api-access-dzvj2") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "kube-api-access-dzvj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:20:28.050288 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.050233 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:20:28.050351 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.050306 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:20:28.050541 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.050499 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:28.050723 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.050699 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:20:28.050809 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.050751 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df8b68b-a7ea-42fb-be97-914415067532-config-out" (OuterVolumeSpecName: "config-out") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:20:28.052149 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.052124 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:28.052785 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.052751 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df8b68b-a7ea-42fb-be97-914415067532-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:20:28.053159 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.053125 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:28.055860 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.053861 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:28.055860 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.054045 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:28.055860 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.055183 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:28.065151 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.065121 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-web-config" (OuterVolumeSpecName: "web-config") pod "0df8b68b-a7ea-42fb-be97-914415067532" (UID: "0df8b68b-a7ea-42fb-be97-914415067532"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:20:28.147962 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.147920 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.147962 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.147955 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0df8b68b-a7ea-42fb-be97-914415067532-config-out\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.147962 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.147964 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.147975 2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-config\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.147983 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.147993 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.148002 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0df8b68b-a7ea-42fb-be97-914415067532-tls-assets\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.148010 2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-metrics-client-certs\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.148019 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.148028 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.148038 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.148047 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.148056 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzvj2\" (UniqueName: \"kubernetes.io/projected/0df8b68b-a7ea-42fb-be97-914415067532-kube-api-access-dzvj2\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.148065 2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-kube-rbac-proxy\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.148074 2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.148082 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-web-config\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.148092 2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0df8b68b-a7ea-42fb-be97-914415067532-secret-grpc-tls\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.148099 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-metrics-client-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.148107 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df8b68b-a7ea-42fb-be97-914415067532-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.148172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.148115 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0df8b68b-a7ea-42fb-be97-914415067532-prometheus-k8s-db\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:20:28.767412 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.767372 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" event={"ID":"628bdc9e-cc5a-41e9-be17-520d10f0a4f6","Type":"ContainerStarted","Data":"c7071dd2fa2e83570a3f2d00800977676fce5c17ea71412f5b76940e9f0b5930"} Jul 22 09:20:28.770917 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.770887 2564 generic.go:356] "Generic (PLEG): container finished" podID="0df8b68b-a7ea-42fb-be97-914415067532" containerID="23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc" exitCode=0 Jul 22 09:20:28.770917 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.770918 2564 generic.go:356] "Generic (PLEG): container finished" podID="0df8b68b-a7ea-42fb-be97-914415067532" containerID="d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5" exitCode=0 Jul 22 09:20:28.771117 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.770945 2564 generic.go:356] "Generic (PLEG): container finished" podID="0df8b68b-a7ea-42fb-be97-914415067532" containerID="b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477" exitCode=0 Jul 22 09:20:28.771117 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.770955 2564 generic.go:356] "Generic (PLEG): container finished" podID="0df8b68b-a7ea-42fb-be97-914415067532" containerID="791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4" exitCode=0 Jul 22 09:20:28.771117 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.770963 2564 generic.go:356] "Generic (PLEG): container finished" podID="0df8b68b-a7ea-42fb-be97-914415067532" containerID="1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766" exitCode=0 Jul 22 09:20:28.771117 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.770978 2564 generic.go:356] "Generic (PLEG): container finished" podID="0df8b68b-a7ea-42fb-be97-914415067532" containerID="0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4" exitCode=0 Jul 22 09:20:28.771117 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.770955 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerDied","Data":"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc"} Jul 22 09:20:28.771117 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.771003 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.771117 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.771058 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerDied","Data":"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5"} Jul 22 09:20:28.771117 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.771088 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerDied","Data":"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477"} Jul 22 09:20:28.771117 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.771105 2564 scope.go:117] "RemoveContainer" containerID="23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc" Jul 22 09:20:28.771510 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.771128 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerDied","Data":"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4"} Jul 22 09:20:28.771510 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.771168 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerDied","Data":"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766"} Jul 22 09:20:28.771510 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.771208 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerDied","Data":"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4"} Jul 22 09:20:28.771510 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.771249 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0df8b68b-a7ea-42fb-be97-914415067532","Type":"ContainerDied","Data":"cd97b36bd369fafc63d96e3e5c265126fdebfed52bff108477f38a828684128f"} Jul 22 09:20:28.781089 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.780928 2564 scope.go:117] "RemoveContainer" containerID="d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5" Jul 22 09:20:28.789719 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.789704 2564 scope.go:117] "RemoveContainer" containerID="b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477" Jul 22 09:20:28.796928 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.796908 2564 scope.go:117] "RemoveContainer" containerID="791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4" Jul 22 09:20:28.800125 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.800103 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 09:20:28.805172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.805150 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 09:20:28.805720 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.805693 2564 scope.go:117] "RemoveContainer" containerID="1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766" Jul 22 09:20:28.813153 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.813136 2564 scope.go:117] "RemoveContainer" containerID="0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4" Jul 22 09:20:28.821622 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.821598 2564 scope.go:117] "RemoveContainer" containerID="fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d" Jul 22 09:20:28.829872 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.829846 2564 scope.go:117] "RemoveContainer" containerID="23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc" Jul 22 09:20:28.830212 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:20:28.830187 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc\": container with ID starting with 23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc not found: ID does not exist" containerID="23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc" Jul 22 09:20:28.830317 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.830219 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc"} err="failed to get container status \"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc\": rpc error: code = NotFound desc = could not find container \"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc\": container with ID starting with 23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc not found: ID does not exist" Jul 22 09:20:28.830317 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.830240 2564 scope.go:117] "RemoveContainer" containerID="d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5" Jul 22 09:20:28.830482 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:20:28.830459 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5\": container with ID starting with d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5 not found: ID does not exist" containerID="d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5" Jul 22 09:20:28.830594 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.830489 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5"} err="failed to get container status \"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5\": rpc error: code = NotFound desc = could not find container \"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5\": container with ID starting with d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5 not found: ID does not exist" Jul 22 09:20:28.830594 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.830513 2564 scope.go:117] "RemoveContainer" containerID="b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477" Jul 22 09:20:28.830773 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:20:28.830751 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477\": container with ID starting with b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477 not found: ID does not exist" containerID="b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477" Jul 22 09:20:28.830847 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.830781 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477"} err="failed to get container status \"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477\": rpc error: code = NotFound desc = could not find container \"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477\": container with ID starting with b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477 not found: ID does not exist" Jul 22 09:20:28.830847 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.830802 2564 scope.go:117] "RemoveContainer" containerID="791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4" Jul 22 09:20:28.831067 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:20:28.831044 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4\": container with ID starting with 791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4 not found: ID does not exist" containerID="791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4" Jul 22 09:20:28.831135 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.831073 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4"} err="failed to get container status \"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4\": rpc error: code = NotFound desc = could not find container \"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4\": container with ID starting with 791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4 not found: ID does not exist" Jul 22 09:20:28.831135 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.831093 2564 scope.go:117] "RemoveContainer" containerID="1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766" Jul 22 09:20:28.831460 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:20:28.831438 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766\": container with ID starting with 1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766 not found: ID does not exist" containerID="1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766" Jul 22 09:20:28.831546 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.831465 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766"} err="failed to get container status \"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766\": rpc error: code = NotFound desc = could not find container \"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766\": container with ID starting with 1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766 not found: ID does not exist" Jul 22 09:20:28.831546 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.831480 2564 scope.go:117] "RemoveContainer" containerID="0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4" Jul 22 09:20:28.831911 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:20:28.831884 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4\": container with ID starting with 0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4 not found: ID does not exist" containerID="0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4" Jul 22 09:20:28.832067 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.831916 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4"} err="failed to get container status \"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4\": rpc error: code = NotFound desc = could not find container \"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4\": container with ID starting with 0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4 not found: ID does not exist" Jul 22 09:20:28.832556 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.832517 2564 scope.go:117] "RemoveContainer" containerID="fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d" Jul 22 09:20:28.832896 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:20:28.832845 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d\": container with ID starting with fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d not found: ID does not exist" containerID="fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d" Jul 22 09:20:28.832896 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.832878 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d"} err="failed to get container status \"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d\": rpc error: code = NotFound desc = could not find container \"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d\": container with ID starting with fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d not found: ID does not exist" Jul 22 09:20:28.833065 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.832898 2564 scope.go:117] "RemoveContainer" containerID="23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc" Jul 22 09:20:28.833181 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.833163 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc"} err="failed to get container status \"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc\": rpc error: code = NotFound desc = could not find container \"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc\": container with ID starting with 23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc not found: ID does not exist" Jul 22 09:20:28.833246 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.833183 2564 scope.go:117] "RemoveContainer" containerID="d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5" Jul 22 09:20:28.833476 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.833446 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5"} err="failed to get container status \"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5\": rpc error: code = NotFound desc = could not find container \"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5\": container with ID starting with d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5 not found: ID does not exist" Jul 22 09:20:28.833476 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.833470 2564 scope.go:117] "RemoveContainer" containerID="b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477" Jul 22 09:20:28.833717 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.833694 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477"} err="failed to get container status \"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477\": rpc error: code = NotFound desc = could not find container \"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477\": container with ID starting with b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477 not found: ID does not exist" Jul 22 09:20:28.833810 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.833720 2564 scope.go:117] "RemoveContainer" containerID="791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4" Jul 22 09:20:28.833978 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.833930 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4"} err="failed to get container status \"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4\": rpc error: code = NotFound desc = could not find container \"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4\": container with ID starting with 791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4 not found: ID does not exist" Jul 22 09:20:28.833978 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.833977 2564 scope.go:117] "RemoveContainer" containerID="1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766" Jul 22 09:20:28.834237 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.834216 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766"} err="failed to get container status \"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766\": rpc error: code = NotFound desc = could not find container \"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766\": container with ID starting with 1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766 not found: ID does not exist" Jul 22 09:20:28.834237 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.834236 2564 scope.go:117] "RemoveContainer" containerID="0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4" Jul 22 09:20:28.834513 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.834485 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4"} err="failed to get container status \"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4\": rpc error: code = NotFound desc = could not find container \"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4\": container with ID starting with 0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4 not found: ID does not exist" Jul 22 09:20:28.834513 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.834506 2564 scope.go:117] "RemoveContainer" containerID="fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d" Jul 22 09:20:28.834724 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.834703 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 09:20:28.834793 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.834723 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d"} err="failed to get container status \"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d\": rpc error: code = NotFound desc = could not find container \"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d\": container with ID starting with fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d not found: ID does not exist" Jul 22 09:20:28.834793 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.834745 2564 scope.go:117] "RemoveContainer" containerID="23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc" Jul 22 09:20:28.835048 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835027 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc"} err="failed to get container status \"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc\": rpc error: code = NotFound desc = could not find container \"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc\": container with ID starting with 23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc not found: ID does not exist" Jul 22 09:20:28.835048 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835046 2564 scope.go:117] "RemoveContainer" containerID="d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5" Jul 22 09:20:28.835185 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835151 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="thanos-sidecar" Jul 22 09:20:28.835185 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835168 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="thanos-sidecar" Jul 22 09:20:28.835285 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835190 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="prometheus" Jul 22 09:20:28.835285 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835199 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="prometheus" Jul 22 09:20:28.835285 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835219 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="init-config-reloader" Jul 22 09:20:28.835285 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835227 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="init-config-reloader" Jul 22 09:20:28.835285 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835237 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="kube-rbac-proxy-thanos" Jul 22 09:20:28.835285 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835246 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="kube-rbac-proxy-thanos" Jul 22 09:20:28.835285 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835268 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="config-reloader" Jul 22 09:20:28.835285 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835276 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="config-reloader" Jul 22 09:20:28.835285 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835288 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="kube-rbac-proxy" Jul 22 09:20:28.835763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835297 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="kube-rbac-proxy" Jul 22 09:20:28.835763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835309 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="kube-rbac-proxy-web" Jul 22 09:20:28.835763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835317 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="kube-rbac-proxy-web" Jul 22 09:20:28.835763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835329 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5"} err="failed to get container status \"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5\": rpc error: code = NotFound desc = could not find container \"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5\": container with ID starting with d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5 not found: ID does not exist" Jul 22 09:20:28.835763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835349 2564 scope.go:117] "RemoveContainer" containerID="b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477" Jul 22 09:20:28.835763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835378 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="prometheus" Jul 22 09:20:28.835763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835390 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="kube-rbac-proxy-web" Jul 22 09:20:28.835763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835399 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="kube-rbac-proxy" Jul 22 09:20:28.835763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835408 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="thanos-sidecar" Jul 22 09:20:28.835763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835421 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="kube-rbac-proxy-thanos" Jul 22 09:20:28.835763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835431 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0df8b68b-a7ea-42fb-be97-914415067532" containerName="config-reloader" Jul 22 09:20:28.835763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835586 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477"} err="failed to get container status \"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477\": rpc error: code = NotFound desc = could not find container \"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477\": container with ID starting with b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477 not found: ID does not exist" Jul 22 09:20:28.835763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835602 2564 scope.go:117] "RemoveContainer" containerID="791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4" Jul 22 09:20:28.836294 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835822 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4"} err="failed to get container status \"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4\": rpc error: code = NotFound desc = could not find container \"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4\": container with ID starting with 791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4 not found: ID does not exist" Jul 22 09:20:28.836294 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.835839 2564 scope.go:117] "RemoveContainer" containerID="1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766" Jul 22 09:20:28.836294 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.836133 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766"} err="failed to get container status \"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766\": rpc error: code = NotFound desc = could not find container \"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766\": container with ID starting with 1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766 not found: ID does not exist" Jul 22 09:20:28.836294 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.836158 2564 scope.go:117] "RemoveContainer" containerID="0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4" Jul 22 09:20:28.836571 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.836395 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4"} err="failed to get container status \"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4\": rpc error: code = NotFound desc = could not find container \"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4\": container with ID starting with 0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4 not found: ID does not exist" Jul 22 09:20:28.836571 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.836417 2564 scope.go:117] "RemoveContainer" containerID="fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d" Jul 22 09:20:28.836672 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.836636 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d"} err="failed to get container status \"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d\": rpc error: code = NotFound desc = could not find container \"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d\": container with ID starting with fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d not found: ID does not exist" Jul 22 09:20:28.836672 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.836659 2564 scope.go:117] "RemoveContainer" containerID="23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc" Jul 22 09:20:28.836862 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.836842 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc"} err="failed to get container status \"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc\": rpc error: code = NotFound desc = could not find container \"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc\": container with ID starting with 23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc not found: ID does not exist" Jul 22 09:20:28.836923 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.836863 2564 scope.go:117] "RemoveContainer" containerID="d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5" Jul 22 09:20:28.837263 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.837165 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5"} err="failed to get container status \"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5\": rpc error: code = NotFound desc = could not find container \"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5\": container with ID starting with d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5 not found: ID does not exist" Jul 22 09:20:28.837263 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.837190 2564 scope.go:117] "RemoveContainer" containerID="b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477" Jul 22 09:20:28.837551 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.837502 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477"} err="failed to get container status \"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477\": rpc error: code = NotFound desc = could not find container \"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477\": container with ID starting with b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477 not found: ID does not exist" Jul 22 09:20:28.837551 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.837527 2564 scope.go:117] "RemoveContainer" containerID="791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4" Jul 22 09:20:28.837897 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.837871 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4"} err="failed to get container status \"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4\": rpc error: code = NotFound desc = could not find container \"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4\": container with ID starting with 791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4 not found: ID does not exist" Jul 22 09:20:28.838014 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.837898 2564 scope.go:117] "RemoveContainer" containerID="1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766" Jul 22 09:20:28.838402 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.838369 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766"} err="failed to get container status \"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766\": rpc error: code = NotFound desc = could not find container \"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766\": container with ID starting with 1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766 not found: ID does not exist" Jul 22 09:20:28.838402 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.838396 2564 scope.go:117] "RemoveContainer" containerID="0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4" Jul 22 09:20:28.838675 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.838650 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4"} err="failed to get container status \"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4\": rpc error: code = NotFound desc = could not find container \"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4\": container with ID starting with 0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4 not found: ID does not exist" Jul 22 09:20:28.838762 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.838678 2564 scope.go:117] "RemoveContainer" containerID="fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d" Jul 22 09:20:28.838963 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.838919 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d"} err="failed to get container status \"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d\": rpc error: code = NotFound desc = could not find container \"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d\": container with ID starting with fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d not found: ID does not exist" Jul 22 09:20:28.839357 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.839339 2564 scope.go:117] "RemoveContainer" containerID="23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc" Jul 22 09:20:28.839623 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.839605 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.839763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.839716 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc"} err="failed to get container status \"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc\": rpc error: code = NotFound desc = could not find container \"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc\": container with ID starting with 23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc not found: ID does not exist" Jul 22 09:20:28.839763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.839738 2564 scope.go:117] "RemoveContainer" containerID="d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5" Jul 22 09:20:28.840332 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.840302 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5"} err="failed to get container status \"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5\": rpc error: code = NotFound desc = could not find container \"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5\": container with ID starting with d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5 not found: ID does not exist" Jul 22 09:20:28.840398 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.840337 2564 scope.go:117] "RemoveContainer" containerID="b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477" Jul 22 09:20:28.840662 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.840635 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477"} err="failed to get container status \"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477\": rpc error: code = NotFound desc = could not find container \"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477\": container with ID starting with b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477 not found: ID does not exist" Jul 22 09:20:28.840737 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.840665 2564 scope.go:117] "RemoveContainer" containerID="791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4" Jul 22 09:20:28.841118 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.841093 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4"} err="failed to get container status \"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4\": rpc error: code = NotFound desc = could not find container \"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4\": container with ID starting with 791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4 not found: ID does not exist" Jul 22 09:20:28.841205 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.841131 2564 scope.go:117] "RemoveContainer" containerID="1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766" Jul 22 09:20:28.841419 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.841391 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766"} err="failed to get container status \"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766\": rpc error: code = NotFound desc = could not find container \"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766\": container with ID starting with 1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766 not found: ID does not exist" Jul 22 09:20:28.841493 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.841423 2564 scope.go:117] "RemoveContainer" containerID="0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4" Jul 22 09:20:28.841701 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.841676 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4"} err="failed to get container status \"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4\": rpc error: code = NotFound desc = could not find container \"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4\": container with ID starting with 0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4 not found: ID does not exist" Jul 22 09:20:28.841777 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.841704 2564 scope.go:117] "RemoveContainer" containerID="fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d" Jul 22 09:20:28.842000 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.841981 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d"} err="failed to get container status \"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d\": rpc error: code = NotFound desc = could not find container \"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d\": container with ID starting with fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d not found: ID does not exist" Jul 22 09:20:28.842000 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.842000 2564 scope.go:117] "RemoveContainer" containerID="23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc" Jul 22 09:20:28.842226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.842197 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc"} err="failed to get container status \"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc\": rpc error: code = NotFound desc = could not find container \"23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc\": container with ID starting with 23f534fdd0b667564eff8245a3b7142ab20801d36a5053c3f92e0319375334cc not found: ID does not exist" Jul 22 09:20:28.842226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.842220 2564 scope.go:117] "RemoveContainer" containerID="d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5" Jul 22 09:20:28.842561 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.842526 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5"} err="failed to get container status \"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5\": rpc error: code = NotFound desc = could not find container \"d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5\": container with ID starting with d2ee0837a400102a284829f6c2852f7f4480d6fef95df362d92fa45573c5abb5 not found: ID does not exist" Jul 22 09:20:28.842561 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.842546 2564 scope.go:117] "RemoveContainer" containerID="b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477" Jul 22 09:20:28.842811 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.842788 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477"} err="failed to get container status \"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477\": rpc error: code = NotFound desc = could not find container \"b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477\": container with ID starting with b15ba8075cefe0d1bd2c6f61756553e532f2dd0adcb9a296b46c447cb93c9477 not found: ID does not exist" Jul 22 09:20:28.842890 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.842813 2564 scope.go:117] "RemoveContainer" containerID="791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4" Jul 22 09:20:28.843089 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.843052 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4"} err="failed to get container status \"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4\": rpc error: code = NotFound desc = could not find container \"791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4\": container with ID starting with 791fafc27482cd784df525efbbdf5f1881518a6289228463cc25506e356350a4 not found: ID does not exist" Jul 22 09:20:28.843089 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.843089 2564 scope.go:117] "RemoveContainer" containerID="1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766" Jul 22 09:20:28.843246 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.843122 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 22 09:20:28.843246 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.843122 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 22 09:20:28.843371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.843332 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766"} err="failed to get container status \"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766\": rpc error: code = NotFound desc = could not find container \"1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766\": container with ID starting with 1781a16debfa1a9bf288d8176b91f1f0fa4365e5a15017ed3ec055cff08cc766 not found: ID does not exist" Jul 22 09:20:28.843371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.843357 2564 scope.go:117] "RemoveContainer" containerID="0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4" Jul 22 09:20:28.843623 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.843599 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4"} err="failed to get container status \"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4\": rpc error: code = NotFound desc = could not find container \"0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4\": container with ID starting with 0273d257b7c2f86a9c73fa720672da04750bda07c8480cc99aa1061a88046ec4 not found: ID does not exist" Jul 22 09:20:28.843711 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.843625 2564 scope.go:117] "RemoveContainer" containerID="fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d" Jul 22 09:20:28.843903 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.843869 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d"} err="failed to get container status \"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d\": rpc error: code = NotFound desc = could not find container \"fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d\": container with ID starting with fded0aba52add0486a996c68062c92a781c1b75ee00f3e8533cbd8b8529bce2d not found: ID does not exist" Jul 22 09:20:28.844676 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.844196 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 22 09:20:28.844676 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.844229 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 22 09:20:28.844676 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.844378 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 22 09:20:28.844676 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.844569 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zdvcr\"" Jul 22 09:20:28.845281 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.845234 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f8jr6t3povq0a\"" Jul 22 09:20:28.845502 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.845484 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 22 09:20:28.846758 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.846061 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 22 09:20:28.846758 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.846128 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 22 09:20:28.846758 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.846262 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 22 09:20:28.846758 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.846404 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 22 09:20:28.846758 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.846573 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 22 09:20:28.847130 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.846834 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 22 09:20:28.848550 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.848493 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 22 09:20:28.851098 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.851062 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 22 09:20:28.854257 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854236 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854346 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854346 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854312 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854346 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854333 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-web-config\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854463 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854463 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854538 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854459 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854538 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854486 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854538 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854680 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854543 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854680 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854603 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4fa27dbe-6518-441c-bdcb-284d2377fce2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854680 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854651 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854814 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854693 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4fa27dbe-6518-441c-bdcb-284d2377fce2-config-out\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854814 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854719 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854814 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854759 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-config\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854814 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854789 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854998 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854827 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4fa27dbe-6518-441c-bdcb-284d2377fce2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854998 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854846 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854998 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.854998 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.854901 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2k9p\" (UniqueName: \"kubernetes.io/projected/4fa27dbe-6518-441c-bdcb-284d2377fce2-kube-api-access-m2k9p\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.859464 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.859438 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 09:20:28.955903 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.955869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-config\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.956092 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.955909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.956092 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.955964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4fa27dbe-6518-441c-bdcb-284d2377fce2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.956211 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.956139 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.956211 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.956194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.956318 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.956232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2k9p\" (UniqueName: \"kubernetes.io/projected/4fa27dbe-6518-441c-bdcb-284d2377fce2-kube-api-access-m2k9p\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.956318 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.956271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.956318 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.956296 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.956467 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.956332 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4fa27dbe-6518-441c-bdcb-284d2377fce2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.956467 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.956343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.957570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.956977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.957570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.957030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-web-config\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.957570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.957070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.957570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.957097 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.957570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.957143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.957570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.957170 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.957570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.957194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.957570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.957249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.957570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.957278 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4fa27dbe-6518-441c-bdcb-284d2377fce2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.957570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.957303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.957570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.957332 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4fa27dbe-6518-441c-bdcb-284d2377fce2-config-out\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.957570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.957366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.957570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.957533 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.958307 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.958232 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.958868 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.958843 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.959263 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.959120 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.959263 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.959145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.959263 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.959126 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.960498 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.959758 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.960498 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.960235 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.960650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.960495 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4fa27dbe-6518-441c-bdcb-284d2377fce2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.961734 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.961676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.961828 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.961753 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4fa27dbe-6518-441c-bdcb-284d2377fce2-config-out\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.961898 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.961823 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4fa27dbe-6518-441c-bdcb-284d2377fce2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.962878 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.962832 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.963389 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.963366 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-config\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.963658 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.963637 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-web-config\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.963735 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.963656 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.964010 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.963991 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fa27dbe-6518-441c-bdcb-284d2377fce2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:28.966782 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:28.966754 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2k9p\" (UniqueName: \"kubernetes.io/projected/4fa27dbe-6518-441c-bdcb-284d2377fce2-kube-api-access-m2k9p\") pod \"prometheus-k8s-0\" (UID: \"4fa27dbe-6518-441c-bdcb-284d2377fce2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:29.152578 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:29.152530 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:29.318477 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:29.318455 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 22 09:20:29.321271 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:20:29.321241 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa27dbe_6518_441c_bdcb_284d2377fce2.slice/crio-0047c823b81c6bc310056ac8822a45f096d44f6b60896796c17fdb59e19b69c5 WatchSource:0}: Error finding container 0047c823b81c6bc310056ac8822a45f096d44f6b60896796c17fdb59e19b69c5: Status 404 returned error can't find the container with id 0047c823b81c6bc310056ac8822a45f096d44f6b60896796c17fdb59e19b69c5 Jul 22 09:20:29.778891 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:29.778853 2564 generic.go:356] "Generic (PLEG): container finished" podID="4fa27dbe-6518-441c-bdcb-284d2377fce2" containerID="944d5bb41eec94401fb1f8e06e969bf8cce5f99410d70e3632a0e59252a4b72e" exitCode=0 Jul 22 09:20:29.779323 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:29.778901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fa27dbe-6518-441c-bdcb-284d2377fce2","Type":"ContainerDied","Data":"944d5bb41eec94401fb1f8e06e969bf8cce5f99410d70e3632a0e59252a4b72e"} Jul 22 09:20:29.779323 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:29.778973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fa27dbe-6518-441c-bdcb-284d2377fce2","Type":"ContainerStarted","Data":"0047c823b81c6bc310056ac8822a45f096d44f6b60896796c17fdb59e19b69c5"} Jul 22 09:20:30.019546 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:30.019517 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0df8b68b-a7ea-42fb-be97-914415067532" path="/var/lib/kubelet/pods/0df8b68b-a7ea-42fb-be97-914415067532/volumes" Jul 22 09:20:30.785364 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:30.785316 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" event={"ID":"628bdc9e-cc5a-41e9-be17-520d10f0a4f6","Type":"ContainerStarted","Data":"45ab750984b4ef432d8d724f818e6ce68f7304aa57b9daf3cd89601f709ffe38"} Jul 22 09:20:30.785828 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:30.785390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" event={"ID":"628bdc9e-cc5a-41e9-be17-520d10f0a4f6","Type":"ContainerStarted","Data":"7599488269fe190ef842ba0a0043c83b617ec6be260b12178bad4de30c6134be"} Jul 22 09:20:30.785828 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:30.785417 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" event={"ID":"628bdc9e-cc5a-41e9-be17-520d10f0a4f6","Type":"ContainerStarted","Data":"0bb96f2f7013070fed17946ba351a996e3bedd4cd5f55b1dba7b4165a46f5be1"} Jul 22 09:20:30.788372 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:30.788349 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fa27dbe-6518-441c-bdcb-284d2377fce2","Type":"ContainerStarted","Data":"4eb9ad7c463dc90a8745145f39b1f738e614abbdf83ac8f449863afa8ab24f16"} Jul 22 09:20:30.788497 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:30.788413 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fa27dbe-6518-441c-bdcb-284d2377fce2","Type":"ContainerStarted","Data":"524b39cb0e6c0b01d39e8b062bb34c8a4be2cd797eedeec379364e99b98a2caa"} Jul 22 09:20:30.788497 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:30.788451 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fa27dbe-6518-441c-bdcb-284d2377fce2","Type":"ContainerStarted","Data":"7cb2950ad5d0660a94aeb655c52cd8d74cb49df1ba8b03f736cf9a644fc43a73"} Jul 22 09:20:30.788497 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:30.788493 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fa27dbe-6518-441c-bdcb-284d2377fce2","Type":"ContainerStarted","Data":"40c6833008c5c85f3f5b2cbad6cd681257af7611da8eb3075fa844f24bccbbc2"} Jul 22 09:20:30.788661 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:30.788531 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fa27dbe-6518-441c-bdcb-284d2377fce2","Type":"ContainerStarted","Data":"90bfc6c94ad0099da0c3c14f72190a294a3cc56623c4ecc267b28a6c51f34237"} Jul 22 09:20:30.788661 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:30.788576 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4fa27dbe-6518-441c-bdcb-284d2377fce2","Type":"ContainerStarted","Data":"ed44b3acba95e3230c7c3f628e0932a7c2657e2a268952d48f2d39fc665eb966"} Jul 22 09:20:30.820789 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:30.820747 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-59b5558bb7-dvc2r" podStartSLOduration=1.839077744 podStartE2EDuration="3.820732406s" totalImagesPullingTime="1.981654662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:20:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:20:30.819376949 +0000 UTC m=+155.316884388" watchObservedRunningTime="2026-07-22 09:20:30.820732406 +0000 UTC m=+155.318239842" Jul 22 09:20:30.856127 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:30.856074 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.831243517 podStartE2EDuration="2.856059416s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.815899ms" isStatefulPod=false podCreationTimestamp="2026-07-22 09:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:20:30.85352794 +0000 UTC m=+155.351035379" watchObservedRunningTime="2026-07-22 09:20:30.856059416 +0000 UTC m=+155.353566898" Jul 22 09:20:31.628421 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.628387 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cbd587f94-8nbxp"] Jul 22 09:20:31.631739 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.631720 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.647627 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.647603 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cbd587f94-8nbxp"] Jul 22 09:20:31.681733 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.681697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-trusted-ca-bundle\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.681897 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.681754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-service-ca\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.681897 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.681843 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-console-config\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.681897 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.681873 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9jxb\" (UniqueName: \"kubernetes.io/projected/1fab9946-de5f-45d9-840b-c38b3a253b45-kube-api-access-w9jxb\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.682052 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.681911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fab9946-de5f-45d9-840b-c38b3a253b45-console-serving-cert\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.682052 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.681978 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1fab9946-de5f-45d9-840b-c38b3a253b45-console-oauth-config\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.682052 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.682008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-oauth-serving-cert\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.782434 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.782400 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fab9946-de5f-45d9-840b-c38b3a253b45-console-serving-cert\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.782593 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.782450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1fab9946-de5f-45d9-840b-c38b3a253b45-console-oauth-config\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.782593 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.782475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-oauth-serving-cert\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.782593 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.782502 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-trusted-ca-bundle\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.782593 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.782554 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-service-ca\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.782764 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.782602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-console-config\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.782764 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.782626 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9jxb\" (UniqueName: \"kubernetes.io/projected/1fab9946-de5f-45d9-840b-c38b3a253b45-kube-api-access-w9jxb\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.783387 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.783351 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-oauth-serving-cert\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.783512 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.783481 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-trusted-ca-bundle\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.783512 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.783504 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-console-config\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.783835 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.783807 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-service-ca\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.786838 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.786813 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1fab9946-de5f-45d9-840b-c38b3a253b45-console-oauth-config\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.787142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.787034 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fab9946-de5f-45d9-840b-c38b3a253b45-console-serving-cert\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.794710 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.794690 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9jxb\" (UniqueName: \"kubernetes.io/projected/1fab9946-de5f-45d9-840b-c38b3a253b45-kube-api-access-w9jxb\") pod \"console-5cbd587f94-8nbxp\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:31.940814 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:31.940780 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:32.088967 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:32.088930 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cbd587f94-8nbxp"] Jul 22 09:20:32.091027 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:20:32.091000 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fab9946_de5f_45d9_840b_c38b3a253b45.slice/crio-9a58b8387abff68afb9692a68b4400e874f717b9a9e6f11697b4f6809daeda5c WatchSource:0}: Error finding container 9a58b8387abff68afb9692a68b4400e874f717b9a9e6f11697b4f6809daeda5c: Status 404 returned error can't find the container with id 9a58b8387abff68afb9692a68b4400e874f717b9a9e6f11697b4f6809daeda5c Jul 22 09:20:32.799440 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:32.799408 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cbd587f94-8nbxp" event={"ID":"1fab9946-de5f-45d9-840b-c38b3a253b45","Type":"ContainerStarted","Data":"03e5791f42bc2e3de15ce9b53f34ad81a6bf3d1970070c05392311a67a182853"} Jul 22 09:20:32.799840 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:32.799457 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cbd587f94-8nbxp" event={"ID":"1fab9946-de5f-45d9-840b-c38b3a253b45","Type":"ContainerStarted","Data":"9a58b8387abff68afb9692a68b4400e874f717b9a9e6f11697b4f6809daeda5c"} Jul 22 09:20:32.824659 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:32.824614 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cbd587f94-8nbxp" podStartSLOduration=1.824601027 podStartE2EDuration="1.824601027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:20:32.823709235 +0000 UTC m=+157.321216671" watchObservedRunningTime="2026-07-22 09:20:32.824601027 +0000 UTC m=+157.322108462" Jul 22 09:20:34.152876 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:34.152846 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:20:41.941802 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:41.941764 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:41.942228 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:41.941839 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:41.946493 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:41.946472 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:42.843460 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:42.843425 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:20:42.902170 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:20:42.902140 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c486685d6-7gzss"] Jul 22 09:21:07.922800 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:07.922761 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c486685d6-7gzss" podUID="9edf9893-482b-48a8-9aa0-7baf080b74f1" containerName="console" containerID="cri-o://e077bba082ec2e7b780ff6224a650ce9bc42f3ab4feb25fcda89bf7563cefd62" gracePeriod=15 Jul 22 09:21:08.159056 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.159035 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:21:08.211090 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.211061 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-config\") pod \"9edf9893-482b-48a8-9aa0-7baf080b74f1\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " Jul 22 09:21:08.211249 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.211101 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-oauth-serving-cert\") pod \"9edf9893-482b-48a8-9aa0-7baf080b74f1\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " Jul 22 09:21:08.211249 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.211127 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-serving-cert\") pod \"9edf9893-482b-48a8-9aa0-7baf080b74f1\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " Jul 22 09:21:08.211249 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.211166 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csg6m\" (UniqueName: \"kubernetes.io/projected/9edf9893-482b-48a8-9aa0-7baf080b74f1-kube-api-access-csg6m\") pod \"9edf9893-482b-48a8-9aa0-7baf080b74f1\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " Jul 22 09:21:08.211249 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.211238 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-oauth-config\") pod \"9edf9893-482b-48a8-9aa0-7baf080b74f1\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " Jul 22 09:21:08.211458 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.211269 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-service-ca\") pod \"9edf9893-482b-48a8-9aa0-7baf080b74f1\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " Jul 22 09:21:08.211458 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.211294 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-trusted-ca-bundle\") pod \"9edf9893-482b-48a8-9aa0-7baf080b74f1\" (UID: \"9edf9893-482b-48a8-9aa0-7baf080b74f1\") " Jul 22 09:21:08.211458 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.211428 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-config" (OuterVolumeSpecName: "console-config") pod "9edf9893-482b-48a8-9aa0-7baf080b74f1" (UID: "9edf9893-482b-48a8-9aa0-7baf080b74f1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:21:08.211608 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.211476 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9edf9893-482b-48a8-9aa0-7baf080b74f1" (UID: "9edf9893-482b-48a8-9aa0-7baf080b74f1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:21:08.211715 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.211687 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-service-ca" (OuterVolumeSpecName: "service-ca") pod "9edf9893-482b-48a8-9aa0-7baf080b74f1" (UID: "9edf9893-482b-48a8-9aa0-7baf080b74f1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:21:08.211775 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.211747 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-config\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:21:08.211775 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.211758 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9edf9893-482b-48a8-9aa0-7baf080b74f1" (UID: "9edf9893-482b-48a8-9aa0-7baf080b74f1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:21:08.211775 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.211769 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-oauth-serving-cert\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:21:08.213393 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.213372 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edf9893-482b-48a8-9aa0-7baf080b74f1-kube-api-access-csg6m" (OuterVolumeSpecName: "kube-api-access-csg6m") pod "9edf9893-482b-48a8-9aa0-7baf080b74f1" (UID: "9edf9893-482b-48a8-9aa0-7baf080b74f1"). InnerVolumeSpecName "kube-api-access-csg6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:21:08.213393 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.213377 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9edf9893-482b-48a8-9aa0-7baf080b74f1" (UID: "9edf9893-482b-48a8-9aa0-7baf080b74f1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:21:08.213508 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.213431 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9edf9893-482b-48a8-9aa0-7baf080b74f1" (UID: "9edf9893-482b-48a8-9aa0-7baf080b74f1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:21:08.312997 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.312961 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csg6m\" (UniqueName: \"kubernetes.io/projected/9edf9893-482b-48a8-9aa0-7baf080b74f1-kube-api-access-csg6m\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:21:08.312997 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.312995 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-oauth-config\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:21:08.312997 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.313005 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-service-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:21:08.313183 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.313014 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9edf9893-482b-48a8-9aa0-7baf080b74f1-trusted-ca-bundle\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:21:08.313183 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.313023 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9edf9893-482b-48a8-9aa0-7baf080b74f1-console-serving-cert\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:21:08.939917 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.939881 2564 generic.go:356] "Generic (PLEG): container finished" podID="9edf9893-482b-48a8-9aa0-7baf080b74f1" containerID="e077bba082ec2e7b780ff6224a650ce9bc42f3ab4feb25fcda89bf7563cefd62" exitCode=0 Jul 22 09:21:08.940315 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.939947 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c486685d6-7gzss" Jul 22 09:21:08.940315 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.939961 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c486685d6-7gzss" event={"ID":"9edf9893-482b-48a8-9aa0-7baf080b74f1","Type":"ContainerDied","Data":"e077bba082ec2e7b780ff6224a650ce9bc42f3ab4feb25fcda89bf7563cefd62"} Jul 22 09:21:08.940315 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.940011 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c486685d6-7gzss" event={"ID":"9edf9893-482b-48a8-9aa0-7baf080b74f1","Type":"ContainerDied","Data":"2ace0bb253d6ecadebc934b89c18aec09d45cfaf5a3fab8bc0902c2d54e5ae60"} Jul 22 09:21:08.940315 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.940031 2564 scope.go:117] "RemoveContainer" containerID="e077bba082ec2e7b780ff6224a650ce9bc42f3ab4feb25fcda89bf7563cefd62" Jul 22 09:21:08.948325 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.948308 2564 scope.go:117] "RemoveContainer" containerID="e077bba082ec2e7b780ff6224a650ce9bc42f3ab4feb25fcda89bf7563cefd62" Jul 22 09:21:08.948586 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:21:08.948569 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e077bba082ec2e7b780ff6224a650ce9bc42f3ab4feb25fcda89bf7563cefd62\": container with ID starting with e077bba082ec2e7b780ff6224a650ce9bc42f3ab4feb25fcda89bf7563cefd62 not found: ID does not exist" containerID="e077bba082ec2e7b780ff6224a650ce9bc42f3ab4feb25fcda89bf7563cefd62" Jul 22 09:21:08.948642 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.948598 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e077bba082ec2e7b780ff6224a650ce9bc42f3ab4feb25fcda89bf7563cefd62"} err="failed to get container status \"e077bba082ec2e7b780ff6224a650ce9bc42f3ab4feb25fcda89bf7563cefd62\": rpc error: code = NotFound desc = could not find container \"e077bba082ec2e7b780ff6224a650ce9bc42f3ab4feb25fcda89bf7563cefd62\": container with ID starting with e077bba082ec2e7b780ff6224a650ce9bc42f3ab4feb25fcda89bf7563cefd62 not found: ID does not exist" Jul 22 09:21:08.963658 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.963636 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c486685d6-7gzss"] Jul 22 09:21:08.972113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:08.972094 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c486685d6-7gzss"] Jul 22 09:21:10.014039 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:10.013998 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9edf9893-482b-48a8-9aa0-7baf080b74f1" path="/var/lib/kubelet/pods/9edf9893-482b-48a8-9aa0-7baf080b74f1/volumes" Jul 22 09:21:29.153004 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:29.152952 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:21:29.168121 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:29.168098 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:21:30.041785 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:30.041753 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 22 09:21:44.186664 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.186632 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9975b7675-jsb4r"] Jul 22 09:21:44.187112 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.187089 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9edf9893-482b-48a8-9aa0-7baf080b74f1" containerName="console" Jul 22 09:21:44.187112 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.187109 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9edf9893-482b-48a8-9aa0-7baf080b74f1" containerName="console" Jul 22 09:21:44.187234 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.187164 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9edf9893-482b-48a8-9aa0-7baf080b74f1" containerName="console" Jul 22 09:21:44.190720 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.190698 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.203510 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.203489 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9975b7675-jsb4r"] Jul 22 09:21:44.236600 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.236573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-oauth-serving-cert\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.236704 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.236613 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/363c3221-8d92-425b-ba3c-4c8310e04a21-console-oauth-config\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.236704 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.236634 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzk5k\" (UniqueName: \"kubernetes.io/projected/363c3221-8d92-425b-ba3c-4c8310e04a21-kube-api-access-jzk5k\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.236704 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.236659 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-service-ca\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.236704 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.236682 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-console-config\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.236835 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.236725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/363c3221-8d92-425b-ba3c-4c8310e04a21-console-serving-cert\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.236835 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.236738 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-trusted-ca-bundle\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.337812 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.337770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/363c3221-8d92-425b-ba3c-4c8310e04a21-console-serving-cert\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.337812 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.337810 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-trusted-ca-bundle\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.338080 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.337843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-oauth-serving-cert\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.338080 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.337878 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/363c3221-8d92-425b-ba3c-4c8310e04a21-console-oauth-config\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.338080 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.337902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzk5k\" (UniqueName: \"kubernetes.io/projected/363c3221-8d92-425b-ba3c-4c8310e04a21-kube-api-access-jzk5k\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.338080 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.337923 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-service-ca\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.338080 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.337983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-console-config\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.338985 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.338957 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-service-ca\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.338985 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.338965 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-oauth-serving-cert\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.339172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.338997 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-console-config\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.339172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.339018 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-trusted-ca-bundle\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.345062 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.341229 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/363c3221-8d92-425b-ba3c-4c8310e04a21-console-serving-cert\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.345062 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.341259 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/363c3221-8d92-425b-ba3c-4c8310e04a21-console-oauth-config\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.347335 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.347309 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzk5k\" (UniqueName: \"kubernetes.io/projected/363c3221-8d92-425b-ba3c-4c8310e04a21-kube-api-access-jzk5k\") pod \"console-9975b7675-jsb4r\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.499568 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.499469 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:44.627540 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:44.627510 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9975b7675-jsb4r"] Jul 22 09:21:44.630304 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:21:44.630277 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363c3221_8d92_425b_ba3c_4c8310e04a21.slice/crio-d7f207e155938cc3f0e4b092805f7b9897163097cf70d8bd3a174b799d806b19 WatchSource:0}: Error finding container d7f207e155938cc3f0e4b092805f7b9897163097cf70d8bd3a174b799d806b19: Status 404 returned error can't find the container with id d7f207e155938cc3f0e4b092805f7b9897163097cf70d8bd3a174b799d806b19 Jul 22 09:21:45.083625 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:45.083591 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9975b7675-jsb4r" event={"ID":"363c3221-8d92-425b-ba3c-4c8310e04a21","Type":"ContainerStarted","Data":"7480deadb73067241e6aa34bcdc84e04ebb19efae9e53b44a9143dccb2bf435e"} Jul 22 09:21:45.083793 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:45.083639 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9975b7675-jsb4r" event={"ID":"363c3221-8d92-425b-ba3c-4c8310e04a21","Type":"ContainerStarted","Data":"d7f207e155938cc3f0e4b092805f7b9897163097cf70d8bd3a174b799d806b19"} Jul 22 09:21:45.107714 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:45.107666 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9975b7675-jsb4r" podStartSLOduration=1.107652745 podStartE2EDuration="1.107652745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:21:45.105416739 +0000 UTC m=+229.602924176" watchObservedRunningTime="2026-07-22 09:21:45.107652745 +0000 UTC m=+229.605160181" Jul 22 09:21:54.499760 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:54.499707 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:54.500205 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:54.499786 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:54.504903 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:54.504881 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:55.131805 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:55.131778 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:21:55.192400 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:21:55.192371 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cbd587f94-8nbxp"] Jul 22 09:22:20.212855 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.212754 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cbd587f94-8nbxp" podUID="1fab9946-de5f-45d9-840b-c38b3a253b45" containerName="console" containerID="cri-o://03e5791f42bc2e3de15ce9b53f34ad81a6bf3d1970070c05392311a67a182853" gracePeriod=15 Jul 22 09:22:20.447254 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.447232 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:22:20.455727 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.455709 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-console-config\") pod \"1fab9946-de5f-45d9-840b-c38b3a253b45\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " Jul 22 09:22:20.455821 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.455737 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-oauth-serving-cert\") pod \"1fab9946-de5f-45d9-840b-c38b3a253b45\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " Jul 22 09:22:20.455821 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.455758 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1fab9946-de5f-45d9-840b-c38b3a253b45-console-oauth-config\") pod \"1fab9946-de5f-45d9-840b-c38b3a253b45\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " Jul 22 09:22:20.455821 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.455790 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9jxb\" (UniqueName: \"kubernetes.io/projected/1fab9946-de5f-45d9-840b-c38b3a253b45-kube-api-access-w9jxb\") pod \"1fab9946-de5f-45d9-840b-c38b3a253b45\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " Jul 22 09:22:20.455821 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.455811 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-trusted-ca-bundle\") pod \"1fab9946-de5f-45d9-840b-c38b3a253b45\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " Jul 22 09:22:20.456044 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.455837 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-service-ca\") pod \"1fab9946-de5f-45d9-840b-c38b3a253b45\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " Jul 22 09:22:20.456044 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.455860 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fab9946-de5f-45d9-840b-c38b3a253b45-console-serving-cert\") pod \"1fab9946-de5f-45d9-840b-c38b3a253b45\" (UID: \"1fab9946-de5f-45d9-840b-c38b3a253b45\") " Jul 22 09:22:20.456186 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.456164 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1fab9946-de5f-45d9-840b-c38b3a253b45" (UID: "1fab9946-de5f-45d9-840b-c38b3a253b45"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:22:20.456374 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.456342 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1fab9946-de5f-45d9-840b-c38b3a253b45" (UID: "1fab9946-de5f-45d9-840b-c38b3a253b45"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:22:20.456434 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.456361 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-console-config" (OuterVolumeSpecName: "console-config") pod "1fab9946-de5f-45d9-840b-c38b3a253b45" (UID: "1fab9946-de5f-45d9-840b-c38b3a253b45"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:22:20.456434 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.456411 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-service-ca" (OuterVolumeSpecName: "service-ca") pod "1fab9946-de5f-45d9-840b-c38b3a253b45" (UID: "1fab9946-de5f-45d9-840b-c38b3a253b45"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:22:20.457855 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.457836 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fab9946-de5f-45d9-840b-c38b3a253b45-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1fab9946-de5f-45d9-840b-c38b3a253b45" (UID: "1fab9946-de5f-45d9-840b-c38b3a253b45"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:22:20.457970 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.457929 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fab9946-de5f-45d9-840b-c38b3a253b45-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1fab9946-de5f-45d9-840b-c38b3a253b45" (UID: "1fab9946-de5f-45d9-840b-c38b3a253b45"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:22:20.458031 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.458003 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fab9946-de5f-45d9-840b-c38b3a253b45-kube-api-access-w9jxb" (OuterVolumeSpecName: "kube-api-access-w9jxb") pod "1fab9946-de5f-45d9-840b-c38b3a253b45" (UID: "1fab9946-de5f-45d9-840b-c38b3a253b45"). InnerVolumeSpecName "kube-api-access-w9jxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:22:20.557053 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.556965 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9jxb\" (UniqueName: \"kubernetes.io/projected/1fab9946-de5f-45d9-840b-c38b3a253b45-kube-api-access-w9jxb\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:22:20.557053 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.556991 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-trusted-ca-bundle\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:22:20.557053 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.557001 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-service-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:22:20.557053 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.557009 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1fab9946-de5f-45d9-840b-c38b3a253b45-console-serving-cert\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:22:20.557053 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.557018 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-console-config\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:22:20.557053 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.557025 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1fab9946-de5f-45d9-840b-c38b3a253b45-oauth-serving-cert\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:22:20.557053 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:20.557035 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1fab9946-de5f-45d9-840b-c38b3a253b45-console-oauth-config\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:22:21.233675 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:21.233642 2564 generic.go:356] "Generic (PLEG): container finished" podID="1fab9946-de5f-45d9-840b-c38b3a253b45" containerID="03e5791f42bc2e3de15ce9b53f34ad81a6bf3d1970070c05392311a67a182853" exitCode=0 Jul 22 09:22:21.234168 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:21.233687 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cbd587f94-8nbxp" Jul 22 09:22:21.234168 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:21.233733 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cbd587f94-8nbxp" event={"ID":"1fab9946-de5f-45d9-840b-c38b3a253b45","Type":"ContainerDied","Data":"03e5791f42bc2e3de15ce9b53f34ad81a6bf3d1970070c05392311a67a182853"} Jul 22 09:22:21.234168 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:21.233793 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cbd587f94-8nbxp" event={"ID":"1fab9946-de5f-45d9-840b-c38b3a253b45","Type":"ContainerDied","Data":"9a58b8387abff68afb9692a68b4400e874f717b9a9e6f11697b4f6809daeda5c"} Jul 22 09:22:21.234168 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:21.233825 2564 scope.go:117] "RemoveContainer" containerID="03e5791f42bc2e3de15ce9b53f34ad81a6bf3d1970070c05392311a67a182853" Jul 22 09:22:21.242306 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:21.242290 2564 scope.go:117] "RemoveContainer" containerID="03e5791f42bc2e3de15ce9b53f34ad81a6bf3d1970070c05392311a67a182853" Jul 22 09:22:21.242547 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:22:21.242530 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e5791f42bc2e3de15ce9b53f34ad81a6bf3d1970070c05392311a67a182853\": container with ID starting with 03e5791f42bc2e3de15ce9b53f34ad81a6bf3d1970070c05392311a67a182853 not found: ID does not exist" containerID="03e5791f42bc2e3de15ce9b53f34ad81a6bf3d1970070c05392311a67a182853" Jul 22 09:22:21.242608 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:21.242555 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e5791f42bc2e3de15ce9b53f34ad81a6bf3d1970070c05392311a67a182853"} err="failed to get container status \"03e5791f42bc2e3de15ce9b53f34ad81a6bf3d1970070c05392311a67a182853\": rpc error: code = NotFound desc = could not find container \"03e5791f42bc2e3de15ce9b53f34ad81a6bf3d1970070c05392311a67a182853\": container with ID starting with 03e5791f42bc2e3de15ce9b53f34ad81a6bf3d1970070c05392311a67a182853 not found: ID does not exist" Jul 22 09:22:21.257283 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:21.257264 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cbd587f94-8nbxp"] Jul 22 09:22:21.264955 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:21.264922 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cbd587f94-8nbxp"] Jul 22 09:22:22.013326 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:22.013293 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fab9946-de5f-45d9-840b-c38b3a253b45" path="/var/lib/kubelet/pods/1fab9946-de5f-45d9-840b-c38b3a253b45/volumes" Jul 22 09:22:55.874798 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:55.874772 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:22:55.876343 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:55.876321 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:22:55.878419 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:22:55.878406 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 09:23:10.039096 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.039063 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-29k7l"] Jul 22 09:23:10.040898 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.039442 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fab9946-de5f-45d9-840b-c38b3a253b45" containerName="console" Jul 22 09:23:10.040898 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.039455 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fab9946-de5f-45d9-840b-c38b3a253b45" containerName="console" Jul 22 09:23:10.040898 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.039520 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fab9946-de5f-45d9-840b-c38b3a253b45" containerName="console" Jul 22 09:23:10.042109 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.042093 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-29k7l" Jul 22 09:23:10.044884 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.044865 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 09:23:10.046260 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.046241 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 09:23:10.046371 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.046248 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-tj55h\"" Jul 22 09:23:10.055775 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.055754 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-29k7l"] Jul 22 09:23:10.066568 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.066544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/465cffd0-e0b5-47cb-a8cd-0e1962c030ab-tmp\") pod \"openshift-lws-operator-fd8ccff4c-29k7l\" (UID: \"465cffd0-e0b5-47cb-a8cd-0e1962c030ab\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-29k7l" Jul 22 09:23:10.066801 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.066782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jcz8\" (UniqueName: \"kubernetes.io/projected/465cffd0-e0b5-47cb-a8cd-0e1962c030ab-kube-api-access-8jcz8\") pod \"openshift-lws-operator-fd8ccff4c-29k7l\" (UID: \"465cffd0-e0b5-47cb-a8cd-0e1962c030ab\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-29k7l" Jul 22 09:23:10.168037 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.167998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/465cffd0-e0b5-47cb-a8cd-0e1962c030ab-tmp\") pod \"openshift-lws-operator-fd8ccff4c-29k7l\" (UID: \"465cffd0-e0b5-47cb-a8cd-0e1962c030ab\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-29k7l" Jul 22 09:23:10.168221 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.168138 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jcz8\" (UniqueName: \"kubernetes.io/projected/465cffd0-e0b5-47cb-a8cd-0e1962c030ab-kube-api-access-8jcz8\") pod \"openshift-lws-operator-fd8ccff4c-29k7l\" (UID: \"465cffd0-e0b5-47cb-a8cd-0e1962c030ab\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-29k7l" Jul 22 09:23:10.168496 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.168476 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/465cffd0-e0b5-47cb-a8cd-0e1962c030ab-tmp\") pod \"openshift-lws-operator-fd8ccff4c-29k7l\" (UID: \"465cffd0-e0b5-47cb-a8cd-0e1962c030ab\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-29k7l" Jul 22 09:23:10.177101 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.177078 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jcz8\" (UniqueName: \"kubernetes.io/projected/465cffd0-e0b5-47cb-a8cd-0e1962c030ab-kube-api-access-8jcz8\") pod \"openshift-lws-operator-fd8ccff4c-29k7l\" (UID: \"465cffd0-e0b5-47cb-a8cd-0e1962c030ab\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-29k7l" Jul 22 09:23:10.360270 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.360189 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-29k7l" Jul 22 09:23:10.485758 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.485683 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-29k7l"] Jul 22 09:23:10.487907 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:23:10.487881 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod465cffd0_e0b5_47cb_a8cd_0e1962c030ab.slice/crio-8eb21d8e81ec1bda4fe7e34135d6d572173bb0dfdf826978f07ec9b0f53d81a9 WatchSource:0}: Error finding container 8eb21d8e81ec1bda4fe7e34135d6d572173bb0dfdf826978f07ec9b0f53d81a9: Status 404 returned error can't find the container with id 8eb21d8e81ec1bda4fe7e34135d6d572173bb0dfdf826978f07ec9b0f53d81a9 Jul 22 09:23:10.489759 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:10.489738 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 09:23:11.435293 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:11.435256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-29k7l" event={"ID":"465cffd0-e0b5-47cb-a8cd-0e1962c030ab","Type":"ContainerStarted","Data":"8eb21d8e81ec1bda4fe7e34135d6d572173bb0dfdf826978f07ec9b0f53d81a9"} Jul 22 09:23:13.445763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:13.445730 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-29k7l" event={"ID":"465cffd0-e0b5-47cb-a8cd-0e1962c030ab","Type":"ContainerStarted","Data":"32d53540a72eede6fae112c8150f2efb55e3d759112233664b24e3f28667b3fd"} Jul 22 09:23:13.464730 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:13.464683 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-29k7l" podStartSLOduration=1.086444271 podStartE2EDuration="3.464667237s" totalImagesPullingTime="2.378222966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:23:13.463694623 +0000 UTC m=+317.961202060" watchObservedRunningTime="2026-07-22 09:23:13.464667237 +0000 UTC m=+317.962174672" Jul 22 09:23:15.563356 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:15.563323 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zjn25"] Jul 22 09:23:15.565879 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:15.565863 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-zjn25" Jul 22 09:23:15.568897 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:15.568869 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 09:23:15.569876 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:15.569859 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gqwnv\"" Jul 22 09:23:15.569969 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:15.569880 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 09:23:15.576959 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:15.576921 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zjn25"] Jul 22 09:23:15.616748 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:15.616724 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpbtt\" (UniqueName: \"kubernetes.io/projected/d71cd802-3e3f-4a2f-a62a-1992fd73a21d-kube-api-access-mpbtt\") pod \"cert-manager-55b4d954d5-zjn25\" (UID: \"d71cd802-3e3f-4a2f-a62a-1992fd73a21d\") " pod="cert-manager/cert-manager-55b4d954d5-zjn25" Jul 22 09:23:15.616870 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:15.616773 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d71cd802-3e3f-4a2f-a62a-1992fd73a21d-bound-sa-token\") pod \"cert-manager-55b4d954d5-zjn25\" (UID: \"d71cd802-3e3f-4a2f-a62a-1992fd73a21d\") " pod="cert-manager/cert-manager-55b4d954d5-zjn25" Jul 22 09:23:15.717971 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:15.717905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbtt\" (UniqueName: \"kubernetes.io/projected/d71cd802-3e3f-4a2f-a62a-1992fd73a21d-kube-api-access-mpbtt\") pod \"cert-manager-55b4d954d5-zjn25\" (UID: \"d71cd802-3e3f-4a2f-a62a-1992fd73a21d\") " pod="cert-manager/cert-manager-55b4d954d5-zjn25" Jul 22 09:23:15.718137 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:15.718000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d71cd802-3e3f-4a2f-a62a-1992fd73a21d-bound-sa-token\") pod \"cert-manager-55b4d954d5-zjn25\" (UID: \"d71cd802-3e3f-4a2f-a62a-1992fd73a21d\") " pod="cert-manager/cert-manager-55b4d954d5-zjn25" Jul 22 09:23:15.727266 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:15.727239 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d71cd802-3e3f-4a2f-a62a-1992fd73a21d-bound-sa-token\") pod \"cert-manager-55b4d954d5-zjn25\" (UID: \"d71cd802-3e3f-4a2f-a62a-1992fd73a21d\") " pod="cert-manager/cert-manager-55b4d954d5-zjn25" Jul 22 09:23:15.727530 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:15.727506 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpbtt\" (UniqueName: \"kubernetes.io/projected/d71cd802-3e3f-4a2f-a62a-1992fd73a21d-kube-api-access-mpbtt\") pod \"cert-manager-55b4d954d5-zjn25\" (UID: \"d71cd802-3e3f-4a2f-a62a-1992fd73a21d\") " pod="cert-manager/cert-manager-55b4d954d5-zjn25" Jul 22 09:23:15.874367 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:15.874283 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-zjn25" Jul 22 09:23:16.005155 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:16.005128 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-zjn25"] Jul 22 09:23:16.007659 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:23:16.007636 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd71cd802_3e3f_4a2f_a62a_1992fd73a21d.slice/crio-33285cbebee422e69a3e343d51fd5bef710015b042a2cb842c9133308b4eace8 WatchSource:0}: Error finding container 33285cbebee422e69a3e343d51fd5bef710015b042a2cb842c9133308b4eace8: Status 404 returned error can't find the container with id 33285cbebee422e69a3e343d51fd5bef710015b042a2cb842c9133308b4eace8 Jul 22 09:23:16.459171 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:16.459141 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-zjn25" event={"ID":"d71cd802-3e3f-4a2f-a62a-1992fd73a21d","Type":"ContainerStarted","Data":"33285cbebee422e69a3e343d51fd5bef710015b042a2cb842c9133308b4eace8"} Jul 22 09:23:19.476027 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:19.475992 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-zjn25" event={"ID":"d71cd802-3e3f-4a2f-a62a-1992fd73a21d","Type":"ContainerStarted","Data":"5b701582dba4dc5c1a69c34d51e87974eedc8b0247d62091b2f1e4ef9c5fd3c1"} Jul 22 09:23:19.495950 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:19.495878 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-zjn25" podStartSLOduration=1.356022535 podStartE2EDuration="4.495857823s" totalImagesPullingTime="3.139835288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:23:19.494108526 +0000 UTC m=+323.991615954" watchObservedRunningTime="2026-07-22 09:23:19.495857823 +0000 UTC m=+323.993365260" Jul 22 09:23:22.668188 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.668153 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2"] Jul 22 09:23:22.670428 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.670409 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:22.674210 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.674185 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 22 09:23:22.674210 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.674204 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 22 09:23:22.674360 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.674221 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 22 09:23:22.674360 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.674337 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nnm2t\"" Jul 22 09:23:22.682485 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.682463 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2"] Jul 22 09:23:22.781550 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.781514 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a-manager-config\") pod \"lws-controller-manager-7d9cd798d9-csxx2\" (UID: \"cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:22.781726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.781555 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscw8\" (UniqueName: \"kubernetes.io/projected/cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a-kube-api-access-kscw8\") pod \"lws-controller-manager-7d9cd798d9-csxx2\" (UID: \"cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:22.781726 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.781671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a-cert\") pod \"lws-controller-manager-7d9cd798d9-csxx2\" (UID: \"cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:22.781830 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.781736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a-metrics-cert\") pod \"lws-controller-manager-7d9cd798d9-csxx2\" (UID: \"cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:22.882286 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.882253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a-metrics-cert\") pod \"lws-controller-manager-7d9cd798d9-csxx2\" (UID: \"cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:22.882426 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.882301 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a-manager-config\") pod \"lws-controller-manager-7d9cd798d9-csxx2\" (UID: \"cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:22.882426 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.882317 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscw8\" (UniqueName: \"kubernetes.io/projected/cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a-kube-api-access-kscw8\") pod \"lws-controller-manager-7d9cd798d9-csxx2\" (UID: \"cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:22.882426 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.882364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a-cert\") pod \"lws-controller-manager-7d9cd798d9-csxx2\" (UID: \"cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:22.883015 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.882993 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a-manager-config\") pod \"lws-controller-manager-7d9cd798d9-csxx2\" (UID: \"cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:22.884683 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.884659 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a-metrics-cert\") pod \"lws-controller-manager-7d9cd798d9-csxx2\" (UID: \"cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:22.884830 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.884807 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a-cert\") pod \"lws-controller-manager-7d9cd798d9-csxx2\" (UID: \"cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:22.892705 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.892683 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscw8\" (UniqueName: \"kubernetes.io/projected/cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a-kube-api-access-kscw8\") pod \"lws-controller-manager-7d9cd798d9-csxx2\" (UID: \"cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a\") " pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:22.979243 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:22.979217 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:23.112332 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:23.112300 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2"] Jul 22 09:23:23.114463 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:23:23.114439 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6bdb8c_4a7c_4a86_8a75_d284f666fa4a.slice/crio-bc9618a5a7ee55e430e918a101e37a3ebec71a6917b198ec36d32914ab872caf WatchSource:0}: Error finding container bc9618a5a7ee55e430e918a101e37a3ebec71a6917b198ec36d32914ab872caf: Status 404 returned error can't find the container with id bc9618a5a7ee55e430e918a101e37a3ebec71a6917b198ec36d32914ab872caf Jul 22 09:23:23.493813 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:23.493770 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" event={"ID":"cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a","Type":"ContainerStarted","Data":"bc9618a5a7ee55e430e918a101e37a3ebec71a6917b198ec36d32914ab872caf"} Jul 22 09:23:25.508558 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:25.508522 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" event={"ID":"cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a","Type":"ContainerStarted","Data":"8197f23c6b6fa5a07b596260a8f34b5b2e2cda292b4198ba7e20a7ca9d779485"} Jul 22 09:23:25.509071 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:25.508662 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:32.778873 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.778816 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" podStartSLOduration=9.280967246 podStartE2EDuration="10.778798282s" totalImagesPullingTime="1.497831036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:23:25.541254556 +0000 UTC m=+330.038761994" watchObservedRunningTime="2026-07-22 09:23:32.778798282 +0000 UTC m=+337.276305719" Jul 22 09:23:32.779386 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.779370 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz"] Jul 22 09:23:32.781903 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.781886 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:32.784537 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.784514 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 09:23:32.784672 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.784611 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 09:23:32.784672 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.784618 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 09:23:32.784672 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.784618 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fx9bz\"" Jul 22 09:23:32.784814 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.784627 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 09:23:32.800579 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.800551 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz"] Jul 22 09:23:32.869792 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.869758 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9ca8253-c53a-4a53-82fb-ef777b42a7e8-manifests\") pod \"opendatahub-operator-controller-manager-67797c6b-jwjxz\" (UID: \"b9ca8253-c53a-4a53-82fb-ef777b42a7e8\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:32.869970 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.869803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9ca8253-c53a-4a53-82fb-ef777b42a7e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67797c6b-jwjxz\" (UID: \"b9ca8253-c53a-4a53-82fb-ef777b42a7e8\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:32.869970 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.869905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9ca8253-c53a-4a53-82fb-ef777b42a7e8-tmp\") pod \"opendatahub-operator-controller-manager-67797c6b-jwjxz\" (UID: \"b9ca8253-c53a-4a53-82fb-ef777b42a7e8\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:32.870045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.869971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9ca8253-c53a-4a53-82fb-ef777b42a7e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-67797c6b-jwjxz\" (UID: \"b9ca8253-c53a-4a53-82fb-ef777b42a7e8\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:32.870045 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.869998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccn27\" (UniqueName: \"kubernetes.io/projected/b9ca8253-c53a-4a53-82fb-ef777b42a7e8-kube-api-access-ccn27\") pod \"opendatahub-operator-controller-manager-67797c6b-jwjxz\" (UID: \"b9ca8253-c53a-4a53-82fb-ef777b42a7e8\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:32.970783 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.970752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9ca8253-c53a-4a53-82fb-ef777b42a7e8-manifests\") pod \"opendatahub-operator-controller-manager-67797c6b-jwjxz\" (UID: \"b9ca8253-c53a-4a53-82fb-ef777b42a7e8\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:32.970783 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.970787 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9ca8253-c53a-4a53-82fb-ef777b42a7e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67797c6b-jwjxz\" (UID: \"b9ca8253-c53a-4a53-82fb-ef777b42a7e8\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:32.971039 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.970908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9ca8253-c53a-4a53-82fb-ef777b42a7e8-tmp\") pod \"opendatahub-operator-controller-manager-67797c6b-jwjxz\" (UID: \"b9ca8253-c53a-4a53-82fb-ef777b42a7e8\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:32.971039 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.970967 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9ca8253-c53a-4a53-82fb-ef777b42a7e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-67797c6b-jwjxz\" (UID: \"b9ca8253-c53a-4a53-82fb-ef777b42a7e8\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:32.971039 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.970992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccn27\" (UniqueName: \"kubernetes.io/projected/b9ca8253-c53a-4a53-82fb-ef777b42a7e8-kube-api-access-ccn27\") pod \"opendatahub-operator-controller-manager-67797c6b-jwjxz\" (UID: \"b9ca8253-c53a-4a53-82fb-ef777b42a7e8\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:32.971201 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.971144 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b9ca8253-c53a-4a53-82fb-ef777b42a7e8-manifests\") pod \"opendatahub-operator-controller-manager-67797c6b-jwjxz\" (UID: \"b9ca8253-c53a-4a53-82fb-ef777b42a7e8\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:32.973118 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.973098 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9ca8253-c53a-4a53-82fb-ef777b42a7e8-tmp\") pod \"opendatahub-operator-controller-manager-67797c6b-jwjxz\" (UID: \"b9ca8253-c53a-4a53-82fb-ef777b42a7e8\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:32.973362 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.973339 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9ca8253-c53a-4a53-82fb-ef777b42a7e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-67797c6b-jwjxz\" (UID: \"b9ca8253-c53a-4a53-82fb-ef777b42a7e8\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:32.973430 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.973416 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b9ca8253-c53a-4a53-82fb-ef777b42a7e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67797c6b-jwjxz\" (UID: \"b9ca8253-c53a-4a53-82fb-ef777b42a7e8\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:32.978357 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:32.978338 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccn27\" (UniqueName: \"kubernetes.io/projected/b9ca8253-c53a-4a53-82fb-ef777b42a7e8-kube-api-access-ccn27\") pod \"opendatahub-operator-controller-manager-67797c6b-jwjxz\" (UID: \"b9ca8253-c53a-4a53-82fb-ef777b42a7e8\") " pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:33.091777 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:33.091691 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:33.227232 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:33.227202 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz"] Jul 22 09:23:33.230407 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:23:33.230381 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ca8253_c53a_4a53_82fb_ef777b42a7e8.slice/crio-e08631369d870cf40b2b17e774e603b91bd78b858130143d0f7afcee13a55d5b WatchSource:0}: Error finding container e08631369d870cf40b2b17e774e603b91bd78b858130143d0f7afcee13a55d5b: Status 404 returned error can't find the container with id e08631369d870cf40b2b17e774e603b91bd78b858130143d0f7afcee13a55d5b Jul 22 09:23:33.542382 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:33.542344 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" event={"ID":"b9ca8253-c53a-4a53-82fb-ef777b42a7e8","Type":"ContainerStarted","Data":"e08631369d870cf40b2b17e774e603b91bd78b858130143d0f7afcee13a55d5b"} Jul 22 09:23:36.515152 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:36.515074 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7d9cd798d9-csxx2" Jul 22 09:23:36.565853 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:36.565815 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" event={"ID":"b9ca8253-c53a-4a53-82fb-ef777b42a7e8","Type":"ContainerStarted","Data":"747e3d585e68d8169c494617716f0dfcfe4e5bb5e81de896e22b11f18089d54b"} Jul 22 09:23:37.571902 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:37.571866 2564 generic.go:356] "Generic (PLEG): container finished" podID="b9ca8253-c53a-4a53-82fb-ef777b42a7e8" containerID="747e3d585e68d8169c494617716f0dfcfe4e5bb5e81de896e22b11f18089d54b" exitCode=0 Jul 22 09:23:37.572279 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:37.571959 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" event={"ID":"b9ca8253-c53a-4a53-82fb-ef777b42a7e8","Type":"ContainerDied","Data":"747e3d585e68d8169c494617716f0dfcfe4e5bb5e81de896e22b11f18089d54b"} Jul 22 09:23:38.577740 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:38.577699 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" event={"ID":"b9ca8253-c53a-4a53-82fb-ef777b42a7e8","Type":"ContainerStarted","Data":"e09e1dafcd74e9662a38c8a96363df965fc247746e37c8a4998dced8ccba5cfa"} Jul 22 09:23:38.578222 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:38.577792 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:38.603871 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:38.603811 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" podStartSLOduration=2.100053575 podStartE2EDuration="6.603788781s" totalImagesPullingTime="3.332753146s" totalInitContainerRuntime="1.17098206s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:23:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:23:38.600123149 +0000 UTC m=+343.097630586" watchObservedRunningTime="2026-07-22 09:23:38.603788781 +0000 UTC m=+343.101296218" Jul 22 09:23:49.584062 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:49.583983 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67797c6b-jwjxz" Jul 22 09:23:51.962855 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:51.962816 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb"] Jul 22 09:23:51.968760 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:51.968729 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb" Jul 22 09:23:51.972216 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:51.972192 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 09:23:51.972354 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:51.972238 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 09:23:51.973246 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:51.973223 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 09:23:51.973383 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:51.973325 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 09:23:51.973383 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:51.973264 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-p9fr4\"" Jul 22 09:23:51.983959 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:51.983912 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb"] Jul 22 09:23:52.138466 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:52.138414 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b405e7-01fd-4676-837b-ade2217d49cc-tls-certs\") pod \"kube-auth-proxy-68755c4d9c-rcxzb\" (UID: \"42b405e7-01fd-4676-837b-ade2217d49cc\") " pod="openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb" Jul 22 09:23:52.138652 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:52.138549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcthq\" (UniqueName: \"kubernetes.io/projected/42b405e7-01fd-4676-837b-ade2217d49cc-kube-api-access-lcthq\") pod \"kube-auth-proxy-68755c4d9c-rcxzb\" (UID: \"42b405e7-01fd-4676-837b-ade2217d49cc\") " pod="openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb" Jul 22 09:23:52.138652 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:52.138591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42b405e7-01fd-4676-837b-ade2217d49cc-tmp\") pod \"kube-auth-proxy-68755c4d9c-rcxzb\" (UID: \"42b405e7-01fd-4676-837b-ade2217d49cc\") " pod="openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb" Jul 22 09:23:52.239333 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:52.239250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcthq\" (UniqueName: \"kubernetes.io/projected/42b405e7-01fd-4676-837b-ade2217d49cc-kube-api-access-lcthq\") pod \"kube-auth-proxy-68755c4d9c-rcxzb\" (UID: \"42b405e7-01fd-4676-837b-ade2217d49cc\") " pod="openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb" Jul 22 09:23:52.239333 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:52.239289 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42b405e7-01fd-4676-837b-ade2217d49cc-tmp\") pod \"kube-auth-proxy-68755c4d9c-rcxzb\" (UID: \"42b405e7-01fd-4676-837b-ade2217d49cc\") " pod="openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb" Jul 22 09:23:52.239333 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:52.239324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b405e7-01fd-4676-837b-ade2217d49cc-tls-certs\") pod \"kube-auth-proxy-68755c4d9c-rcxzb\" (UID: \"42b405e7-01fd-4676-837b-ade2217d49cc\") " pod="openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb" Jul 22 09:23:52.239562 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:23:52.239428 2564 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 22 09:23:52.239562 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:23:52.239486 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b405e7-01fd-4676-837b-ade2217d49cc-tls-certs podName:42b405e7-01fd-4676-837b-ade2217d49cc nodeName:}" failed. No retries permitted until 2026-07-22 09:23:52.739472214 +0000 UTC m=+357.236979628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/42b405e7-01fd-4676-837b-ade2217d49cc-tls-certs") pod "kube-auth-proxy-68755c4d9c-rcxzb" (UID: "42b405e7-01fd-4676-837b-ade2217d49cc") : secret "kube-auth-proxy-tls" not found Jul 22 09:23:52.241654 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:52.241632 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42b405e7-01fd-4676-837b-ade2217d49cc-tmp\") pod \"kube-auth-proxy-68755c4d9c-rcxzb\" (UID: \"42b405e7-01fd-4676-837b-ade2217d49cc\") " pod="openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb" Jul 22 09:23:52.261475 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:52.261440 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcthq\" (UniqueName: \"kubernetes.io/projected/42b405e7-01fd-4676-837b-ade2217d49cc-kube-api-access-lcthq\") pod \"kube-auth-proxy-68755c4d9c-rcxzb\" (UID: \"42b405e7-01fd-4676-837b-ade2217d49cc\") " pod="openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb" Jul 22 09:23:52.744057 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:52.744027 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b405e7-01fd-4676-837b-ade2217d49cc-tls-certs\") pod \"kube-auth-proxy-68755c4d9c-rcxzb\" (UID: \"42b405e7-01fd-4676-837b-ade2217d49cc\") " pod="openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb" Jul 22 09:23:52.746324 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:52.746303 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b405e7-01fd-4676-837b-ade2217d49cc-tls-certs\") pod \"kube-auth-proxy-68755c4d9c-rcxzb\" (UID: \"42b405e7-01fd-4676-837b-ade2217d49cc\") " pod="openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb" Jul 22 09:23:52.882226 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:52.882194 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb" Jul 22 09:23:53.025522 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:53.025312 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb"] Jul 22 09:23:53.646772 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:53.646719 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb" event={"ID":"42b405e7-01fd-4676-837b-ade2217d49cc","Type":"ContainerStarted","Data":"378c2df23aedce4170ccaad5e884511a9caaf30bd55dc22dfa0ef4938017b6d4"} Jul 22 09:23:55.657527 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:55.657498 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb" event={"ID":"42b405e7-01fd-4676-837b-ade2217d49cc","Type":"ContainerStarted","Data":"947d7fd0b5fb8281b1a3719bfa9aa7563e7e500f36ffb6e97424d00263c0ef8f"} Jul 22 09:23:55.686471 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:23:55.686418 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68755c4d9c-rcxzb" podStartSLOduration=2.173349847 podStartE2EDuration="4.68639804s" totalImagesPullingTime="2.513048193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:23:55.684704447 +0000 UTC m=+360.182211885" watchObservedRunningTime="2026-07-22 09:23:55.68639804 +0000 UTC m=+360.183905478" Jul 22 09:25:35.759732 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.759690 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b966bd857-skg5r"] Jul 22 09:25:35.763050 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.763029 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.800126 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.800098 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b966bd857-skg5r"] Jul 22 09:25:35.849990 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.849954 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8939cc7a-a481-4a67-8836-92edc5e15f9a-oauth-serving-cert\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.850154 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.849997 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8939cc7a-a481-4a67-8836-92edc5e15f9a-console-oauth-config\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.850154 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.850047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8939cc7a-a481-4a67-8836-92edc5e15f9a-trusted-ca-bundle\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.850154 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.850120 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8939cc7a-a481-4a67-8836-92edc5e15f9a-console-config\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.850154 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.850149 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbssn\" (UniqueName: \"kubernetes.io/projected/8939cc7a-a481-4a67-8836-92edc5e15f9a-kube-api-access-fbssn\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.850321 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.850170 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8939cc7a-a481-4a67-8836-92edc5e15f9a-service-ca\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.850321 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.850188 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8939cc7a-a481-4a67-8836-92edc5e15f9a-console-serving-cert\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.951334 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.951299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8939cc7a-a481-4a67-8836-92edc5e15f9a-oauth-serving-cert\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.951334 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.951333 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8939cc7a-a481-4a67-8836-92edc5e15f9a-console-oauth-config\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.951576 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.951364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8939cc7a-a481-4a67-8836-92edc5e15f9a-trusted-ca-bundle\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.951576 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.951404 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8939cc7a-a481-4a67-8836-92edc5e15f9a-console-config\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.951576 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.951422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbssn\" (UniqueName: \"kubernetes.io/projected/8939cc7a-a481-4a67-8836-92edc5e15f9a-kube-api-access-fbssn\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.951576 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.951440 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8939cc7a-a481-4a67-8836-92edc5e15f9a-service-ca\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.951576 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.951461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8939cc7a-a481-4a67-8836-92edc5e15f9a-console-serving-cert\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.952230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.952203 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8939cc7a-a481-4a67-8836-92edc5e15f9a-oauth-serving-cert\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.952346 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.952251 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8939cc7a-a481-4a67-8836-92edc5e15f9a-console-config\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.952346 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.952303 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8939cc7a-a481-4a67-8836-92edc5e15f9a-service-ca\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.952540 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.952521 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8939cc7a-a481-4a67-8836-92edc5e15f9a-trusted-ca-bundle\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.953948 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.953913 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8939cc7a-a481-4a67-8836-92edc5e15f9a-console-serving-cert\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.954058 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.954037 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8939cc7a-a481-4a67-8836-92edc5e15f9a-console-oauth-config\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:35.972794 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:35.972777 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbssn\" (UniqueName: \"kubernetes.io/projected/8939cc7a-a481-4a67-8836-92edc5e15f9a-kube-api-access-fbssn\") pod \"console-b966bd857-skg5r\" (UID: \"8939cc7a-a481-4a67-8836-92edc5e15f9a\") " pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:36.072141 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:36.072052 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:36.238620 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:36.238466 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b966bd857-skg5r"] Jul 22 09:25:36.240837 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:25:36.240812 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8939cc7a_a481_4a67_8836_92edc5e15f9a.slice/crio-3c75042714bec14717bf87fe6e6aba00d486becf7980b0ade623ef992a3bb727 WatchSource:0}: Error finding container 3c75042714bec14717bf87fe6e6aba00d486becf7980b0ade623ef992a3bb727: Status 404 returned error can't find the container with id 3c75042714bec14717bf87fe6e6aba00d486becf7980b0ade623ef992a3bb727 Jul 22 09:25:37.103630 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:37.103591 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b966bd857-skg5r" event={"ID":"8939cc7a-a481-4a67-8836-92edc5e15f9a","Type":"ContainerStarted","Data":"c916f6cae97f6f97cfc58c4074d56edd0e663d608909a064d93a9774627ce413"} Jul 22 09:25:37.104026 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:37.103654 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b966bd857-skg5r" event={"ID":"8939cc7a-a481-4a67-8836-92edc5e15f9a","Type":"ContainerStarted","Data":"3c75042714bec14717bf87fe6e6aba00d486becf7980b0ade623ef992a3bb727"} Jul 22 09:25:37.146453 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:37.146402 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b966bd857-skg5r" podStartSLOduration=2.146386786 podStartE2EDuration="2.146386786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:25:37.138884311 +0000 UTC m=+461.636391749" watchObservedRunningTime="2026-07-22 09:25:37.146386786 +0000 UTC m=+461.643894222" Jul 22 09:25:39.492782 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.492747 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw"] Jul 22 09:25:39.496104 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.496089 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw" Jul 22 09:25:39.499421 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.499402 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 22 09:25:39.500852 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.500834 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 09:25:39.501049 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.501031 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wnw8l\"" Jul 22 09:25:39.501777 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.501763 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 22 09:25:39.504548 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.504532 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 09:25:39.514869 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.514848 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw"] Jul 22 09:25:39.589153 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.589124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz8gs\" (UniqueName: \"kubernetes.io/projected/c1d40468-de6d-43e6-9257-b448ca3fc792-kube-api-access-sz8gs\") pod \"kuadrant-console-plugin-66b5774985-v4pxw\" (UID: \"c1d40468-de6d-43e6-9257-b448ca3fc792\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw" Jul 22 09:25:39.589317 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.589265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c1d40468-de6d-43e6-9257-b448ca3fc792-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-v4pxw\" (UID: \"c1d40468-de6d-43e6-9257-b448ca3fc792\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw" Jul 22 09:25:39.589317 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.589298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1d40468-de6d-43e6-9257-b448ca3fc792-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-v4pxw\" (UID: \"c1d40468-de6d-43e6-9257-b448ca3fc792\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw" Jul 22 09:25:39.690093 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.690056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c1d40468-de6d-43e6-9257-b448ca3fc792-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-v4pxw\" (UID: \"c1d40468-de6d-43e6-9257-b448ca3fc792\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw" Jul 22 09:25:39.690293 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.690103 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1d40468-de6d-43e6-9257-b448ca3fc792-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-v4pxw\" (UID: \"c1d40468-de6d-43e6-9257-b448ca3fc792\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw" Jul 22 09:25:39.690293 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.690144 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz8gs\" (UniqueName: \"kubernetes.io/projected/c1d40468-de6d-43e6-9257-b448ca3fc792-kube-api-access-sz8gs\") pod \"kuadrant-console-plugin-66b5774985-v4pxw\" (UID: \"c1d40468-de6d-43e6-9257-b448ca3fc792\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw" Jul 22 09:25:39.690770 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.690744 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c1d40468-de6d-43e6-9257-b448ca3fc792-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-v4pxw\" (UID: \"c1d40468-de6d-43e6-9257-b448ca3fc792\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw" Jul 22 09:25:39.692531 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.692510 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1d40468-de6d-43e6-9257-b448ca3fc792-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-v4pxw\" (UID: \"c1d40468-de6d-43e6-9257-b448ca3fc792\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw" Jul 22 09:25:39.708815 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.708793 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz8gs\" (UniqueName: \"kubernetes.io/projected/c1d40468-de6d-43e6-9257-b448ca3fc792-kube-api-access-sz8gs\") pod \"kuadrant-console-plugin-66b5774985-v4pxw\" (UID: \"c1d40468-de6d-43e6-9257-b448ca3fc792\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw" Jul 22 09:25:39.805733 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.805651 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw" Jul 22 09:25:39.974197 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:39.974167 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw"] Jul 22 09:25:39.977072 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:25:39.977045 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1d40468_de6d_43e6_9257_b448ca3fc792.slice/crio-6100c1ac502b41342a429cd0e540da57296a2406464c934bcf5cd516bc36550c WatchSource:0}: Error finding container 6100c1ac502b41342a429cd0e540da57296a2406464c934bcf5cd516bc36550c: Status 404 returned error can't find the container with id 6100c1ac502b41342a429cd0e540da57296a2406464c934bcf5cd516bc36550c Jul 22 09:25:40.118991 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:40.118898 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw" event={"ID":"c1d40468-de6d-43e6-9257-b448ca3fc792","Type":"ContainerStarted","Data":"6100c1ac502b41342a429cd0e540da57296a2406464c934bcf5cd516bc36550c"} Jul 22 09:25:46.072459 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:46.072419 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:46.072958 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:46.072488 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:46.079654 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:46.079612 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:46.159880 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:46.159848 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b966bd857-skg5r" Jul 22 09:25:46.247748 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:25:46.247714 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9975b7675-jsb4r"] Jul 22 09:26:02.246158 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:02.246123 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw" event={"ID":"c1d40468-de6d-43e6-9257-b448ca3fc792","Type":"ContainerStarted","Data":"c2f48e75b2d447268f256f85fc69c813ec13fd4d1ef197d9e5d7c3dcde084555"} Jul 22 09:26:02.273038 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:02.272971 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-v4pxw" podStartSLOduration=1.79021008 podStartE2EDuration="23.272956454s" totalImagesPullingTime="21.482746374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:26:02.272482767 +0000 UTC m=+486.769990204" watchObservedRunningTime="2026-07-22 09:26:02.272956454 +0000 UTC m=+486.770463886" Jul 22 09:26:11.273036 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.272979 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9975b7675-jsb4r" podUID="363c3221-8d92-425b-ba3c-4c8310e04a21" containerName="console" containerID="cri-o://7480deadb73067241e6aa34bcdc84e04ebb19efae9e53b44a9143dccb2bf435e" gracePeriod=15 Jul 22 09:26:11.516435 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.516412 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:26:11.589249 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.589181 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-service-ca\") pod \"363c3221-8d92-425b-ba3c-4c8310e04a21\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " Jul 22 09:26:11.589249 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.589233 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzk5k\" (UniqueName: \"kubernetes.io/projected/363c3221-8d92-425b-ba3c-4c8310e04a21-kube-api-access-jzk5k\") pod \"363c3221-8d92-425b-ba3c-4c8310e04a21\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " Jul 22 09:26:11.589414 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.589280 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/363c3221-8d92-425b-ba3c-4c8310e04a21-console-serving-cert\") pod \"363c3221-8d92-425b-ba3c-4c8310e04a21\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " Jul 22 09:26:11.589414 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.589342 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-console-config\") pod \"363c3221-8d92-425b-ba3c-4c8310e04a21\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " Jul 22 09:26:11.589414 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.589365 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-trusted-ca-bundle\") pod \"363c3221-8d92-425b-ba3c-4c8310e04a21\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " Jul 22 09:26:11.589414 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.589399 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/363c3221-8d92-425b-ba3c-4c8310e04a21-console-oauth-config\") pod \"363c3221-8d92-425b-ba3c-4c8310e04a21\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " Jul 22 09:26:11.589619 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.589452 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-oauth-serving-cert\") pod \"363c3221-8d92-425b-ba3c-4c8310e04a21\" (UID: \"363c3221-8d92-425b-ba3c-4c8310e04a21\") " Jul 22 09:26:11.589619 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.589579 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-service-ca" (OuterVolumeSpecName: "service-ca") pod "363c3221-8d92-425b-ba3c-4c8310e04a21" (UID: "363c3221-8d92-425b-ba3c-4c8310e04a21"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:26:11.589786 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.589751 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-console-config" (OuterVolumeSpecName: "console-config") pod "363c3221-8d92-425b-ba3c-4c8310e04a21" (UID: "363c3221-8d92-425b-ba3c-4c8310e04a21"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:26:11.589860 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.589808 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-service-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:26:11.589860 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.589842 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "363c3221-8d92-425b-ba3c-4c8310e04a21" (UID: "363c3221-8d92-425b-ba3c-4c8310e04a21"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:26:11.589973 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.589865 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "363c3221-8d92-425b-ba3c-4c8310e04a21" (UID: "363c3221-8d92-425b-ba3c-4c8310e04a21"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:26:11.591470 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.591446 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363c3221-8d92-425b-ba3c-4c8310e04a21-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "363c3221-8d92-425b-ba3c-4c8310e04a21" (UID: "363c3221-8d92-425b-ba3c-4c8310e04a21"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:26:11.591877 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.591852 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363c3221-8d92-425b-ba3c-4c8310e04a21-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "363c3221-8d92-425b-ba3c-4c8310e04a21" (UID: "363c3221-8d92-425b-ba3c-4c8310e04a21"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:26:11.591972 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.591871 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363c3221-8d92-425b-ba3c-4c8310e04a21-kube-api-access-jzk5k" (OuterVolumeSpecName: "kube-api-access-jzk5k") pod "363c3221-8d92-425b-ba3c-4c8310e04a21" (UID: "363c3221-8d92-425b-ba3c-4c8310e04a21"). InnerVolumeSpecName "kube-api-access-jzk5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:26:11.691088 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.691055 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-console-config\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:26:11.691088 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.691080 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-trusted-ca-bundle\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:26:11.691088 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.691090 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/363c3221-8d92-425b-ba3c-4c8310e04a21-console-oauth-config\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:26:11.691301 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.691099 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/363c3221-8d92-425b-ba3c-4c8310e04a21-oauth-serving-cert\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:26:11.691301 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.691108 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzk5k\" (UniqueName: \"kubernetes.io/projected/363c3221-8d92-425b-ba3c-4c8310e04a21-kube-api-access-jzk5k\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:26:11.691301 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:11.691116 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/363c3221-8d92-425b-ba3c-4c8310e04a21-console-serving-cert\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:26:12.297905 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:12.297814 2564 generic.go:356] "Generic (PLEG): container finished" podID="363c3221-8d92-425b-ba3c-4c8310e04a21" containerID="7480deadb73067241e6aa34bcdc84e04ebb19efae9e53b44a9143dccb2bf435e" exitCode=0 Jul 22 09:26:12.297905 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:12.297855 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9975b7675-jsb4r" Jul 22 09:26:12.297905 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:12.297891 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9975b7675-jsb4r" event={"ID":"363c3221-8d92-425b-ba3c-4c8310e04a21","Type":"ContainerDied","Data":"7480deadb73067241e6aa34bcdc84e04ebb19efae9e53b44a9143dccb2bf435e"} Jul 22 09:26:12.298409 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:12.297927 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9975b7675-jsb4r" event={"ID":"363c3221-8d92-425b-ba3c-4c8310e04a21","Type":"ContainerDied","Data":"d7f207e155938cc3f0e4b092805f7b9897163097cf70d8bd3a174b799d806b19"} Jul 22 09:26:12.298409 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:12.297973 2564 scope.go:117] "RemoveContainer" containerID="7480deadb73067241e6aa34bcdc84e04ebb19efae9e53b44a9143dccb2bf435e" Jul 22 09:26:12.306322 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:12.306304 2564 scope.go:117] "RemoveContainer" containerID="7480deadb73067241e6aa34bcdc84e04ebb19efae9e53b44a9143dccb2bf435e" Jul 22 09:26:12.306572 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:26:12.306555 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7480deadb73067241e6aa34bcdc84e04ebb19efae9e53b44a9143dccb2bf435e\": container with ID starting with 7480deadb73067241e6aa34bcdc84e04ebb19efae9e53b44a9143dccb2bf435e not found: ID does not exist" containerID="7480deadb73067241e6aa34bcdc84e04ebb19efae9e53b44a9143dccb2bf435e" Jul 22 09:26:12.306618 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:12.306579 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7480deadb73067241e6aa34bcdc84e04ebb19efae9e53b44a9143dccb2bf435e"} err="failed to get container status \"7480deadb73067241e6aa34bcdc84e04ebb19efae9e53b44a9143dccb2bf435e\": rpc error: code = NotFound desc = could not find container \"7480deadb73067241e6aa34bcdc84e04ebb19efae9e53b44a9143dccb2bf435e\": container with ID starting with 7480deadb73067241e6aa34bcdc84e04ebb19efae9e53b44a9143dccb2bf435e not found: ID does not exist" Jul 22 09:26:12.317389 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:12.317367 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9975b7675-jsb4r"] Jul 22 09:26:12.324505 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:12.324487 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9975b7675-jsb4r"] Jul 22 09:26:14.014556 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:14.014522 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="363c3221-8d92-425b-ba3c-4c8310e04a21" path="/var/lib/kubelet/pods/363c3221-8d92-425b-ba3c-4c8310e04a21/volumes" Jul 22 09:26:23.015985 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.015949 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:26:23.016424 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.016313 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="363c3221-8d92-425b-ba3c-4c8310e04a21" containerName="console" Jul 22 09:26:23.016424 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.016325 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="363c3221-8d92-425b-ba3c-4c8310e04a21" containerName="console" Jul 22 09:26:23.016424 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.016391 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="363c3221-8d92-425b-ba3c-4c8310e04a21" containerName="console" Jul 22 09:26:23.136504 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.136465 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:26:23.136504 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.136511 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:26:23.136717 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.136568 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-5488w" Jul 22 09:26:23.139418 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.139398 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 09:26:23.193534 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.193497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05214e24-d32c-403f-9bbd-73535684630e-config-file\") pod \"limitador-limitador-54d6d988d5-5488w\" (UID: \"05214e24-d32c-403f-9bbd-73535684630e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5488w" Jul 22 09:26:23.193714 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.193546 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt7t9\" (UniqueName: \"kubernetes.io/projected/05214e24-d32c-403f-9bbd-73535684630e-kube-api-access-dt7t9\") pod \"limitador-limitador-54d6d988d5-5488w\" (UID: \"05214e24-d32c-403f-9bbd-73535684630e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5488w" Jul 22 09:26:23.294589 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.294506 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt7t9\" (UniqueName: \"kubernetes.io/projected/05214e24-d32c-403f-9bbd-73535684630e-kube-api-access-dt7t9\") pod \"limitador-limitador-54d6d988d5-5488w\" (UID: \"05214e24-d32c-403f-9bbd-73535684630e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5488w" Jul 22 09:26:23.294725 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.294591 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05214e24-d32c-403f-9bbd-73535684630e-config-file\") pod \"limitador-limitador-54d6d988d5-5488w\" (UID: \"05214e24-d32c-403f-9bbd-73535684630e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5488w" Jul 22 09:26:23.295247 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.295228 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/05214e24-d32c-403f-9bbd-73535684630e-config-file\") pod \"limitador-limitador-54d6d988d5-5488w\" (UID: \"05214e24-d32c-403f-9bbd-73535684630e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5488w" Jul 22 09:26:23.303690 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.303668 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt7t9\" (UniqueName: \"kubernetes.io/projected/05214e24-d32c-403f-9bbd-73535684630e-kube-api-access-dt7t9\") pod \"limitador-limitador-54d6d988d5-5488w\" (UID: \"05214e24-d32c-403f-9bbd-73535684630e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5488w" Jul 22 09:26:23.447316 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.447278 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-5488w" Jul 22 09:26:23.612242 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.612216 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:26:23.614551 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:26:23.614508 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05214e24_d32c_403f_9bbd_73535684630e.slice/crio-359d8257749b675f306f3e2eb3c50485b335215f4f6a9c3a23dbc08061fe663d WatchSource:0}: Error finding container 359d8257749b675f306f3e2eb3c50485b335215f4f6a9c3a23dbc08061fe663d: Status 404 returned error can't find the container with id 359d8257749b675f306f3e2eb3c50485b335215f4f6a9c3a23dbc08061fe663d Jul 22 09:26:23.750029 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.749995 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-dw7cg"] Jul 22 09:26:23.781147 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.781115 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dw7cg"] Jul 22 09:26:23.781309 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.781231 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dw7cg" Jul 22 09:26:23.783832 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.783806 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wqn8n\"" Jul 22 09:26:23.899580 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:23.899505 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffbts\" (UniqueName: \"kubernetes.io/projected/c5c45af8-9c30-453c-b186-6b5bd6361505-kube-api-access-ffbts\") pod \"authorino-75d556bd77-dw7cg\" (UID: \"c5c45af8-9c30-453c-b186-6b5bd6361505\") " pod="kuadrant-system/authorino-75d556bd77-dw7cg" Jul 22 09:26:24.000399 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:24.000358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffbts\" (UniqueName: \"kubernetes.io/projected/c5c45af8-9c30-453c-b186-6b5bd6361505-kube-api-access-ffbts\") pod \"authorino-75d556bd77-dw7cg\" (UID: \"c5c45af8-9c30-453c-b186-6b5bd6361505\") " pod="kuadrant-system/authorino-75d556bd77-dw7cg" Jul 22 09:26:24.011074 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:24.011044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffbts\" (UniqueName: \"kubernetes.io/projected/c5c45af8-9c30-453c-b186-6b5bd6361505-kube-api-access-ffbts\") pod \"authorino-75d556bd77-dw7cg\" (UID: \"c5c45af8-9c30-453c-b186-6b5bd6361505\") " pod="kuadrant-system/authorino-75d556bd77-dw7cg" Jul 22 09:26:24.090884 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:24.090855 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dw7cg" Jul 22 09:26:24.233762 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:24.233729 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dw7cg"] Jul 22 09:26:24.235830 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:26:24.235695 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5c45af8_9c30_453c_b186_6b5bd6361505.slice/crio-c31342d6f3ead4dc95364ae32e5f34a9109534639f33e2e693ca897a54ec66d3 WatchSource:0}: Error finding container c31342d6f3ead4dc95364ae32e5f34a9109534639f33e2e693ca897a54ec66d3: Status 404 returned error can't find the container with id c31342d6f3ead4dc95364ae32e5f34a9109534639f33e2e693ca897a54ec66d3 Jul 22 09:26:24.357010 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:24.356974 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dw7cg" event={"ID":"c5c45af8-9c30-453c-b186-6b5bd6361505","Type":"ContainerStarted","Data":"c31342d6f3ead4dc95364ae32e5f34a9109534639f33e2e693ca897a54ec66d3"} Jul 22 09:26:24.359615 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:24.359583 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-5488w" event={"ID":"05214e24-d32c-403f-9bbd-73535684630e","Type":"ContainerStarted","Data":"359d8257749b675f306f3e2eb3c50485b335215f4f6a9c3a23dbc08061fe663d"} Jul 22 09:26:30.397719 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:30.397676 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dw7cg" event={"ID":"c5c45af8-9c30-453c-b186-6b5bd6361505","Type":"ContainerStarted","Data":"89a5e660c7fa99a90850a84412e2aa2b2e29583f2d3e8219759258ad6eeb6f5a"} Jul 22 09:26:30.399203 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:30.399179 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-5488w" event={"ID":"05214e24-d32c-403f-9bbd-73535684630e","Type":"ContainerStarted","Data":"e59352cc453756c69dab0f6bd07cd3da02561438c5abb96c5bc24a9081a75263"} Jul 22 09:26:30.399319 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:30.399306 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-5488w" Jul 22 09:26:30.419235 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:30.419186 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-dw7cg" podStartSLOduration=1.7363156800000001 podStartE2EDuration="7.419172133s" totalImagesPullingTime="5.682856453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:26:30.417340686 +0000 UTC m=+514.914848120" watchObservedRunningTime="2026-07-22 09:26:30.419172133 +0000 UTC m=+514.916679568" Jul 22 09:26:30.439019 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:30.438971 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-5488w" podStartSLOduration=2.14095569 podStartE2EDuration="8.438955311s" totalImagesPullingTime="6.297999621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:26:30.436676165 +0000 UTC m=+514.934183603" watchObservedRunningTime="2026-07-22 09:26:30.438955311 +0000 UTC m=+514.936462747" Jul 22 09:26:41.405395 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:41.405362 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-5488w" Jul 22 09:26:57.607043 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:57.607016 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-695bb7dcf5-zz459"] Jul 22 09:26:57.611625 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:57.611607 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-695bb7dcf5-zz459" Jul 22 09:26:57.616021 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:57.615999 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 22 09:26:57.621763 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:57.621739 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-695bb7dcf5-zz459"] Jul 22 09:26:57.703964 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:57.703901 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9500e085-2d50-4167-a169-3549c1605ff1-tls-cert\") pod \"authorino-695bb7dcf5-zz459\" (UID: \"9500e085-2d50-4167-a169-3549c1605ff1\") " pod="kuadrant-system/authorino-695bb7dcf5-zz459" Jul 22 09:26:57.704137 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:57.704041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cwpt\" (UniqueName: \"kubernetes.io/projected/9500e085-2d50-4167-a169-3549c1605ff1-kube-api-access-8cwpt\") pod \"authorino-695bb7dcf5-zz459\" (UID: \"9500e085-2d50-4167-a169-3549c1605ff1\") " pod="kuadrant-system/authorino-695bb7dcf5-zz459" Jul 22 09:26:57.804674 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:57.804641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cwpt\" (UniqueName: \"kubernetes.io/projected/9500e085-2d50-4167-a169-3549c1605ff1-kube-api-access-8cwpt\") pod \"authorino-695bb7dcf5-zz459\" (UID: \"9500e085-2d50-4167-a169-3549c1605ff1\") " pod="kuadrant-system/authorino-695bb7dcf5-zz459" Jul 22 09:26:57.804848 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:57.804725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9500e085-2d50-4167-a169-3549c1605ff1-tls-cert\") pod \"authorino-695bb7dcf5-zz459\" (UID: \"9500e085-2d50-4167-a169-3549c1605ff1\") " pod="kuadrant-system/authorino-695bb7dcf5-zz459" Jul 22 09:26:57.807065 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:57.807044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9500e085-2d50-4167-a169-3549c1605ff1-tls-cert\") pod \"authorino-695bb7dcf5-zz459\" (UID: \"9500e085-2d50-4167-a169-3549c1605ff1\") " pod="kuadrant-system/authorino-695bb7dcf5-zz459" Jul 22 09:26:57.813086 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:57.813065 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cwpt\" (UniqueName: \"kubernetes.io/projected/9500e085-2d50-4167-a169-3549c1605ff1-kube-api-access-8cwpt\") pod \"authorino-695bb7dcf5-zz459\" (UID: \"9500e085-2d50-4167-a169-3549c1605ff1\") " pod="kuadrant-system/authorino-695bb7dcf5-zz459" Jul 22 09:26:57.920974 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:57.920878 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-695bb7dcf5-zz459" Jul 22 09:26:58.069553 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:58.069518 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-695bb7dcf5-zz459"] Jul 22 09:26:58.072002 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:26:58.071972 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9500e085_2d50_4167_a169_3549c1605ff1.slice/crio-92049672cc11ac5c20c6bf2fd8bd319473a38faca02fc1033fb2aa62a8935b52 WatchSource:0}: Error finding container 92049672cc11ac5c20c6bf2fd8bd319473a38faca02fc1033fb2aa62a8935b52: Status 404 returned error can't find the container with id 92049672cc11ac5c20c6bf2fd8bd319473a38faca02fc1033fb2aa62a8935b52 Jul 22 09:26:58.537062 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:58.537026 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-695bb7dcf5-zz459" event={"ID":"9500e085-2d50-4167-a169-3549c1605ff1","Type":"ContainerStarted","Data":"2effc8df0fc24b008a465767df4a8cd9671afd7a33689a14163cc559d65812d0"} Jul 22 09:26:58.537249 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:58.537072 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-695bb7dcf5-zz459" event={"ID":"9500e085-2d50-4167-a169-3549c1605ff1","Type":"ContainerStarted","Data":"92049672cc11ac5c20c6bf2fd8bd319473a38faca02fc1033fb2aa62a8935b52"} Jul 22 09:26:58.557870 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:58.557821 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-695bb7dcf5-zz459" podStartSLOduration=1.2066765799999999 podStartE2EDuration="1.557805222s" totalImagesPullingTime="351.128642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:26:58.555329133 +0000 UTC m=+543.052836571" watchObservedRunningTime="2026-07-22 09:26:58.557805222 +0000 UTC m=+543.055312690" Jul 22 09:26:58.599048 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:58.599018 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dw7cg"] Jul 22 09:26:58.599255 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:58.599231 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-dw7cg" podUID="c5c45af8-9c30-453c-b186-6b5bd6361505" containerName="authorino" containerID="cri-o://89a5e660c7fa99a90850a84412e2aa2b2e29583f2d3e8219759258ad6eeb6f5a" gracePeriod=30 Jul 22 09:26:58.846359 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:58.846340 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dw7cg" Jul 22 09:26:58.916613 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:58.916581 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffbts\" (UniqueName: \"kubernetes.io/projected/c5c45af8-9c30-453c-b186-6b5bd6361505-kube-api-access-ffbts\") pod \"c5c45af8-9c30-453c-b186-6b5bd6361505\" (UID: \"c5c45af8-9c30-453c-b186-6b5bd6361505\") " Jul 22 09:26:58.918679 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:58.918653 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c45af8-9c30-453c-b186-6b5bd6361505-kube-api-access-ffbts" (OuterVolumeSpecName: "kube-api-access-ffbts") pod "c5c45af8-9c30-453c-b186-6b5bd6361505" (UID: "c5c45af8-9c30-453c-b186-6b5bd6361505"). InnerVolumeSpecName "kube-api-access-ffbts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:26:59.017694 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:59.017617 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffbts\" (UniqueName: \"kubernetes.io/projected/c5c45af8-9c30-453c-b186-6b5bd6361505-kube-api-access-ffbts\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:26:59.543381 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:59.543348 2564 generic.go:356] "Generic (PLEG): container finished" podID="c5c45af8-9c30-453c-b186-6b5bd6361505" containerID="89a5e660c7fa99a90850a84412e2aa2b2e29583f2d3e8219759258ad6eeb6f5a" exitCode=0 Jul 22 09:26:59.543557 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:59.543391 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-dw7cg" Jul 22 09:26:59.543557 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:59.543428 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dw7cg" event={"ID":"c5c45af8-9c30-453c-b186-6b5bd6361505","Type":"ContainerDied","Data":"89a5e660c7fa99a90850a84412e2aa2b2e29583f2d3e8219759258ad6eeb6f5a"} Jul 22 09:26:59.543557 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:59.543469 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-dw7cg" event={"ID":"c5c45af8-9c30-453c-b186-6b5bd6361505","Type":"ContainerDied","Data":"c31342d6f3ead4dc95364ae32e5f34a9109534639f33e2e693ca897a54ec66d3"} Jul 22 09:26:59.543557 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:59.543485 2564 scope.go:117] "RemoveContainer" containerID="89a5e660c7fa99a90850a84412e2aa2b2e29583f2d3e8219759258ad6eeb6f5a" Jul 22 09:26:59.552117 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:59.552095 2564 scope.go:117] "RemoveContainer" containerID="89a5e660c7fa99a90850a84412e2aa2b2e29583f2d3e8219759258ad6eeb6f5a" Jul 22 09:26:59.552379 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:26:59.552361 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a5e660c7fa99a90850a84412e2aa2b2e29583f2d3e8219759258ad6eeb6f5a\": container with ID starting with 89a5e660c7fa99a90850a84412e2aa2b2e29583f2d3e8219759258ad6eeb6f5a not found: ID does not exist" containerID="89a5e660c7fa99a90850a84412e2aa2b2e29583f2d3e8219759258ad6eeb6f5a" Jul 22 09:26:59.552450 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:59.552391 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a5e660c7fa99a90850a84412e2aa2b2e29583f2d3e8219759258ad6eeb6f5a"} err="failed to get container status \"89a5e660c7fa99a90850a84412e2aa2b2e29583f2d3e8219759258ad6eeb6f5a\": rpc error: code = NotFound desc = could not find container \"89a5e660c7fa99a90850a84412e2aa2b2e29583f2d3e8219759258ad6eeb6f5a\": container with ID starting with 89a5e660c7fa99a90850a84412e2aa2b2e29583f2d3e8219759258ad6eeb6f5a not found: ID does not exist" Jul 22 09:26:59.568156 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:59.568127 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dw7cg"] Jul 22 09:26:59.570964 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:26:59.570945 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-dw7cg"] Jul 22 09:27:00.014439 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:27:00.014406 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5c45af8-9c30-453c-b186-6b5bd6361505" path="/var/lib/kubelet/pods/c5c45af8-9c30-453c-b186-6b5bd6361505/volumes" Jul 22 09:27:41.377157 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:27:41.377120 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:27:55.908129 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:27:55.908099 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:27:55.908699 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:27:55.908155 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:28:00.364985 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:00.364948 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:28:11.948905 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:11.948860 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:28:19.666063 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:19.665985 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:28:38.972049 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:38.972011 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj"] Jul 22 09:28:38.972469 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:38.972440 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5c45af8-9c30-453c-b186-6b5bd6361505" containerName="authorino" Jul 22 09:28:38.972469 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:38.972453 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5c45af8-9c30-453c-b186-6b5bd6361505" containerName="authorino" Jul 22 09:28:38.972545 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:38.972529 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5c45af8-9c30-453c-b186-6b5bd6361505" containerName="authorino" Jul 22 09:28:38.975658 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:38.975643 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:38.980391 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:38.980369 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 09:28:38.980524 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:38.980403 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 09:28:38.980524 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:38.980506 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xgb7q\"" Jul 22 09:28:38.980645 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:38.980564 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 22 09:28:38.988189 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:38.988166 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj"] Jul 22 09:28:39.140402 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.140363 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.140606 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.140421 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmndc\" (UniqueName: \"kubernetes.io/projected/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-kube-api-access-lmndc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.140606 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.140499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.140606 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.140525 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.140606 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.140567 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.140606 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.140591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.241722 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.241639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.241722 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.241691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmndc\" (UniqueName: \"kubernetes.io/projected/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-kube-api-access-lmndc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.241969 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.241733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.241969 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.241758 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.241969 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.241802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.241969 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.241830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.242238 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.242217 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.242295 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.242223 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.242333 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.242291 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.243844 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.243823 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.244113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.244098 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.251137 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.251113 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmndc\" (UniqueName: \"kubernetes.io/projected/c1ad39d9-6992-4fb6-a401-098c6bfa63f8-kube-api-access-lmndc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj\" (UID: \"c1ad39d9-6992-4fb6-a401-098c6bfa63f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.285570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.285544 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:39.418214 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.418191 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj"] Jul 22 09:28:39.420176 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:28:39.420149 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ad39d9_6992_4fb6_a401_098c6bfa63f8.slice/crio-7f6b24ef6cae1955c67991e99f12f01eabfddddcdb0910d82d621a2611fa3655 WatchSource:0}: Error finding container 7f6b24ef6cae1955c67991e99f12f01eabfddddcdb0910d82d621a2611fa3655: Status 404 returned error can't find the container with id 7f6b24ef6cae1955c67991e99f12f01eabfddddcdb0910d82d621a2611fa3655 Jul 22 09:28:39.421487 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:39.421470 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 09:28:40.042570 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:40.042494 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" event={"ID":"c1ad39d9-6992-4fb6-a401-098c6bfa63f8","Type":"ContainerStarted","Data":"7f6b24ef6cae1955c67991e99f12f01eabfddddcdb0910d82d621a2611fa3655"} Jul 22 09:28:40.508230 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:40.508199 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:28:41.049295 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:41.049248 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" event={"ID":"c1ad39d9-6992-4fb6-a401-098c6bfa63f8","Type":"ContainerStarted","Data":"3038617b3e01b369a4a2381d44a40b60502ae1911d5a5fa32315b3c22a2c4b23"} Jul 22 09:28:41.073400 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:41.073352 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" podStartSLOduration=1.8488996260000001 podStartE2EDuration="3.073336607s" totalImagesPullingTime="1.224436981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:28:41.072352886 +0000 UTC m=+645.569860323" watchObservedRunningTime="2026-07-22 09:28:41.073336607 +0000 UTC m=+645.570844043" Jul 22 09:28:49.286228 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:49.286187 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:49.286690 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:49.286269 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:49.298518 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:49.298493 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:28:50.102394 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:28:50.102368 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj" Jul 22 09:29:01.458099 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:01.458065 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:29:04.550254 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.550223 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n"] Jul 22 09:29:04.553871 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.553850 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.556588 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.556566 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 22 09:29:04.565200 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.565173 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n"] Jul 22 09:29:04.666248 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.666210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45f6ea2f-a25e-44f9-b357-2495db4265fe-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.666423 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.666269 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45f6ea2f-a25e-44f9-b357-2495db4265fe-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.666423 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.666321 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpb7g\" (UniqueName: \"kubernetes.io/projected/45f6ea2f-a25e-44f9-b357-2495db4265fe-kube-api-access-lpb7g\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.666423 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.666362 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45f6ea2f-a25e-44f9-b357-2495db4265fe-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.666423 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.666393 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45f6ea2f-a25e-44f9-b357-2495db4265fe-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.666423 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.666419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45f6ea2f-a25e-44f9-b357-2495db4265fe-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.767890 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.767859 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45f6ea2f-a25e-44f9-b357-2495db4265fe-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.768072 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.767902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpb7g\" (UniqueName: \"kubernetes.io/projected/45f6ea2f-a25e-44f9-b357-2495db4265fe-kube-api-access-lpb7g\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.768072 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.767950 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45f6ea2f-a25e-44f9-b357-2495db4265fe-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.768072 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.767980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45f6ea2f-a25e-44f9-b357-2495db4265fe-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.768072 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.768005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45f6ea2f-a25e-44f9-b357-2495db4265fe-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.768072 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.768069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45f6ea2f-a25e-44f9-b357-2495db4265fe-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.768395 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.768373 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45f6ea2f-a25e-44f9-b357-2495db4265fe-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.768466 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.768415 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45f6ea2f-a25e-44f9-b357-2495db4265fe-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.768466 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.768427 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45f6ea2f-a25e-44f9-b357-2495db4265fe-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.770081 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.770066 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45f6ea2f-a25e-44f9-b357-2495db4265fe-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.770530 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.770507 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45f6ea2f-a25e-44f9-b357-2495db4265fe-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.779424 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.779402 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpb7g\" (UniqueName: \"kubernetes.io/projected/45f6ea2f-a25e-44f9-b357-2495db4265fe-kube-api-access-lpb7g\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n\" (UID: \"45f6ea2f-a25e-44f9-b357-2495db4265fe\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.863750 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.863681 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:04.995672 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:04.995647 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n"] Jul 22 09:29:04.998259 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:29:04.998233 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45f6ea2f_a25e_44f9_b357_2495db4265fe.slice/crio-2a10bf741c61fe304f194d2ec7ced054b637425310a6c094cc3788818276d92d WatchSource:0}: Error finding container 2a10bf741c61fe304f194d2ec7ced054b637425310a6c094cc3788818276d92d: Status 404 returned error can't find the container with id 2a10bf741c61fe304f194d2ec7ced054b637425310a6c094cc3788818276d92d Jul 22 09:29:05.168713 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:05.168630 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" event={"ID":"45f6ea2f-a25e-44f9-b357-2495db4265fe","Type":"ContainerStarted","Data":"2a10bf741c61fe304f194d2ec7ced054b637425310a6c094cc3788818276d92d"} Jul 22 09:29:06.175055 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:06.175022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" event={"ID":"45f6ea2f-a25e-44f9-b357-2495db4265fe","Type":"ContainerStarted","Data":"f7cb15a5a41a8834fb7842095d3de42467dcb75000646497a076244a1d49ac66"} Jul 22 09:29:06.196890 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:06.196842 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" podStartSLOduration=2.016974411 podStartE2EDuration="2.196827101s" totalImagesPullingTime="179.85269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:29:06.195653641 +0000 UTC m=+670.693161080" watchObservedRunningTime="2026-07-22 09:29:06.196827101 +0000 UTC m=+670.694334570" Jul 22 09:29:12.958521 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:12.958485 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:29:14.864515 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:14.864478 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:14.865028 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:14.864539 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:14.876727 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:14.876702 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:15.234534 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:15.234509 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n" Jul 22 09:29:58.380689 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:58.380651 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-695bb7dcf5-zz459"] Jul 22 09:29:58.381173 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:58.380884 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-695bb7dcf5-zz459" podUID="9500e085-2d50-4167-a169-3549c1605ff1" containerName="authorino" containerID="cri-o://2effc8df0fc24b008a465767df4a8cd9671afd7a33689a14163cc559d65812d0" gracePeriod=30 Jul 22 09:29:58.632610 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:58.632554 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-695bb7dcf5-zz459" Jul 22 09:29:58.787423 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:58.787389 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cwpt\" (UniqueName: \"kubernetes.io/projected/9500e085-2d50-4167-a169-3549c1605ff1-kube-api-access-8cwpt\") pod \"9500e085-2d50-4167-a169-3549c1605ff1\" (UID: \"9500e085-2d50-4167-a169-3549c1605ff1\") " Jul 22 09:29:58.787610 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:58.787478 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9500e085-2d50-4167-a169-3549c1605ff1-tls-cert\") pod \"9500e085-2d50-4167-a169-3549c1605ff1\" (UID: \"9500e085-2d50-4167-a169-3549c1605ff1\") " Jul 22 09:29:58.789528 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:58.789501 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9500e085-2d50-4167-a169-3549c1605ff1-kube-api-access-8cwpt" (OuterVolumeSpecName: "kube-api-access-8cwpt") pod "9500e085-2d50-4167-a169-3549c1605ff1" (UID: "9500e085-2d50-4167-a169-3549c1605ff1"). InnerVolumeSpecName "kube-api-access-8cwpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:29:58.797316 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:58.797291 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9500e085-2d50-4167-a169-3549c1605ff1-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9500e085-2d50-4167-a169-3549c1605ff1" (UID: "9500e085-2d50-4167-a169-3549c1605ff1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:29:58.889161 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:58.889092 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9500e085-2d50-4167-a169-3549c1605ff1-tls-cert\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:29:58.889161 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:58.889116 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cwpt\" (UniqueName: \"kubernetes.io/projected/9500e085-2d50-4167-a169-3549c1605ff1-kube-api-access-8cwpt\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:29:59.427237 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:59.427203 2564 generic.go:356] "Generic (PLEG): container finished" podID="9500e085-2d50-4167-a169-3549c1605ff1" containerID="2effc8df0fc24b008a465767df4a8cd9671afd7a33689a14163cc559d65812d0" exitCode=0 Jul 22 09:29:59.427599 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:59.427258 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-695bb7dcf5-zz459" Jul 22 09:29:59.427599 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:59.427265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-695bb7dcf5-zz459" event={"ID":"9500e085-2d50-4167-a169-3549c1605ff1","Type":"ContainerDied","Data":"2effc8df0fc24b008a465767df4a8cd9671afd7a33689a14163cc559d65812d0"} Jul 22 09:29:59.427599 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:59.427297 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-695bb7dcf5-zz459" event={"ID":"9500e085-2d50-4167-a169-3549c1605ff1","Type":"ContainerDied","Data":"92049672cc11ac5c20c6bf2fd8bd319473a38faca02fc1033fb2aa62a8935b52"} Jul 22 09:29:59.427599 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:59.427313 2564 scope.go:117] "RemoveContainer" containerID="2effc8df0fc24b008a465767df4a8cd9671afd7a33689a14163cc559d65812d0" Jul 22 09:29:59.436115 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:59.436086 2564 scope.go:117] "RemoveContainer" containerID="2effc8df0fc24b008a465767df4a8cd9671afd7a33689a14163cc559d65812d0" Jul 22 09:29:59.436358 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:29:59.436337 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2effc8df0fc24b008a465767df4a8cd9671afd7a33689a14163cc559d65812d0\": container with ID starting with 2effc8df0fc24b008a465767df4a8cd9671afd7a33689a14163cc559d65812d0 not found: ID does not exist" containerID="2effc8df0fc24b008a465767df4a8cd9671afd7a33689a14163cc559d65812d0" Jul 22 09:29:59.436411 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:59.436366 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2effc8df0fc24b008a465767df4a8cd9671afd7a33689a14163cc559d65812d0"} err="failed to get container status \"2effc8df0fc24b008a465767df4a8cd9671afd7a33689a14163cc559d65812d0\": rpc error: code = NotFound desc = could not find container \"2effc8df0fc24b008a465767df4a8cd9671afd7a33689a14163cc559d65812d0\": container with ID starting with 2effc8df0fc24b008a465767df4a8cd9671afd7a33689a14163cc559d65812d0 not found: ID does not exist" Jul 22 09:29:59.450612 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:59.450592 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-695bb7dcf5-zz459"] Jul 22 09:29:59.458650 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:29:59.458620 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-695bb7dcf5-zz459"] Jul 22 09:30:00.015786 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:30:00.015745 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9500e085-2d50-4167-a169-3549c1605ff1" path="/var/lib/kubelet/pods/9500e085-2d50-4167-a169-3549c1605ff1/volumes" Jul 22 09:30:28.371445 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:30:28.371408 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:30:40.255827 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:30:40.255788 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:30:50.663251 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:30:50.663204 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:32:31.949354 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:32:31.949318 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:32:55.938856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:32:55.938830 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:32:55.939337 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:32:55.939220 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:33:11.851273 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:33:11.851237 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:33:21.449804 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:33:21.449767 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:33:31.956709 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:33:31.956675 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:33:38.153394 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:33:38.153359 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:33:47.557722 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:33:47.557688 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:33:57.350319 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:33:57.350280 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:34:09.752610 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:34:09.752574 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:34:19.551771 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:34:19.551700 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:34:41.252264 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:34:41.252229 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:34:51.984868 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:34:51.984832 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:35:16.668703 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:35:16.668666 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:35:29.452910 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:35:29.452871 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:35:45.159881 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:35:45.159844 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:35:54.456482 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:35:54.456400 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:36:04.066682 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:36:04.066646 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:36:14.452042 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:36:14.452005 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:36:23.894160 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:36:23.894125 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:37:55.969387 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:37:55.969362 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:37:55.970000 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:37:55.969455 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:38:09.656475 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:38:09.656441 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:38:18.876608 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:38:18.876575 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:38:31.466355 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:38:31.466316 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:38:42.664979 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:38:42.664928 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:38:53.759563 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:38:53.759485 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:39:05.656623 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:39:05.656587 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:39:13.668369 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:39:13.668326 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:39:24.845476 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:39:24.845432 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:39:35.857669 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:39:35.857633 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:39:44.357265 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:39:44.357233 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:39:55.266087 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:39:55.266051 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:40:19.554899 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:19.554806 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:40:26.867725 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:26.867690 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq"] Jul 22 09:40:26.868365 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:26.868345 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9500e085-2d50-4167-a169-3549c1605ff1" containerName="authorino" Jul 22 09:40:26.868454 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:26.868367 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9500e085-2d50-4167-a169-3549c1605ff1" containerName="authorino" Jul 22 09:40:26.868510 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:26.868477 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9500e085-2d50-4167-a169-3549c1605ff1" containerName="authorino" Jul 22 09:40:26.871562 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:26.871542 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:26.874115 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:26.874096 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0fe1d095-kserve-self-signed-certs\"" Jul 22 09:40:26.892265 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:26.892233 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq"] Jul 22 09:40:26.957947 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:26.957912 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-dshm\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:26.958098 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:26.957963 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-home\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:26.958098 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:26.958045 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fctz\" (UniqueName: \"kubernetes.io/projected/a241580e-ba27-4fc1-9e43-59d47b941130-kube-api-access-7fctz\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:26.958176 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:26.958101 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-model-cache\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:26.958176 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:26.958171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a241580e-ba27-4fc1-9e43-59d47b941130-tls-certs\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:26.958245 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:26.958189 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-tmp-dir\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:26.969389 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:26.969361 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:40:27.058844 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.058816 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-model-cache\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:27.059013 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.058880 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a241580e-ba27-4fc1-9e43-59d47b941130-tls-certs\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:27.059013 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.058905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-tmp-dir\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:27.059013 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.058952 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-dshm\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:27.059013 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.059003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-home\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:27.059187 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.059047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fctz\" (UniqueName: \"kubernetes.io/projected/a241580e-ba27-4fc1-9e43-59d47b941130-kube-api-access-7fctz\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:27.059302 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.059282 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-model-cache\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:27.059539 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.059511 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-tmp-dir\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:27.059633 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.059601 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-home\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:27.061338 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.061314 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-dshm\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:27.061725 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.061706 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a241580e-ba27-4fc1-9e43-59d47b941130-tls-certs\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:27.067275 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.067258 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fctz\" (UniqueName: \"kubernetes.io/projected/a241580e-ba27-4fc1-9e43-59d47b941130-kube-api-access-7fctz\") pod \"e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:27.181180 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.181111 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:27.313214 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.313184 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq"] Jul 22 09:40:27.315554 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:40:27.315521 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda241580e_ba27_4fc1_9e43_59d47b941130.slice/crio-31d34c181925956b988e378911f9b0c80e2a4ca0d7b0249f827596ce588e9037 WatchSource:0}: Error finding container 31d34c181925956b988e378911f9b0c80e2a4ca0d7b0249f827596ce588e9037: Status 404 returned error can't find the container with id 31d34c181925956b988e378911f9b0c80e2a4ca0d7b0249f827596ce588e9037 Jul 22 09:40:27.389078 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.388901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" event={"ID":"a241580e-ba27-4fc1-9e43-59d47b941130","Type":"ContainerStarted","Data":"bc9760dc3fac7ffee787b5d4b1ec0b076ab193b391f7d815e01b8713a8a2a695"} Jul 22 09:40:27.389078 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.388966 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" event={"ID":"a241580e-ba27-4fc1-9e43-59d47b941130","Type":"ContainerStarted","Data":"31d34c181925956b988e378911f9b0c80e2a4ca0d7b0249f827596ce588e9037"} Jul 22 09:40:27.415475 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:27.415424 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" podStartSLOduration=1.415408667 podStartE2EDuration="1.415408667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:40:27.413577952 +0000 UTC m=+1351.911085390" watchObservedRunningTime="2026-07-22 09:40:27.415408667 +0000 UTC m=+1351.912916102" Jul 22 09:40:37.181455 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:37.181371 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:37.181455 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:37.181424 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:37.193967 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:37.193922 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:37.448422 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:37.448391 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:40:39.255420 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:39.255382 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:40:48.256040 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.255991 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz"] Jul 22 09:40:48.260484 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.260462 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.263308 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.263286 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0fe1d095-kserve-self-signed-certs\"" Jul 22 09:40:48.271879 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.271852 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz"] Jul 22 09:40:48.359594 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.359560 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-model-cache\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.359594 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.359602 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-home\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.359868 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.359624 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-tmp-dir\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.359868 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.359730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-dshm\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.359868 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.359800 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9f63887-e905-4e22-8bf9-19a35d61a5e2-tls-certs\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.360056 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.359917 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zhwf\" (UniqueName: \"kubernetes.io/projected/d9f63887-e905-4e22-8bf9-19a35d61a5e2-kube-api-access-6zhwf\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.460504 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.460463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zhwf\" (UniqueName: \"kubernetes.io/projected/d9f63887-e905-4e22-8bf9-19a35d61a5e2-kube-api-access-6zhwf\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.460671 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.460509 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-model-cache\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.460671 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.460543 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-home\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.460671 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.460569 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-tmp-dir\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.460671 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.460608 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-dshm\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.460671 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.460659 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9f63887-e905-4e22-8bf9-19a35d61a5e2-tls-certs\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.461074 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.461045 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-model-cache\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.461135 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.461089 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-tmp-dir\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.461135 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.461091 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-home\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.462948 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.462911 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-dshm\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.463336 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.463317 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9f63887-e905-4e22-8bf9-19a35d61a5e2-tls-certs\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.468677 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.468655 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zhwf\" (UniqueName: \"kubernetes.io/projected/d9f63887-e905-4e22-8bf9-19a35d61a5e2-kube-api-access-6zhwf\") pod \"e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.571875 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.571780 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:48.707297 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:48.707271 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz"] Jul 22 09:40:48.709354 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:40:48.709323 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9f63887_e905_4e22_8bf9_19a35d61a5e2.slice/crio-9b79795217d741601c85e245bcd58b27a8ac2a7bbf517808ee89b067dbf82272 WatchSource:0}: Error finding container 9b79795217d741601c85e245bcd58b27a8ac2a7bbf517808ee89b067dbf82272: Status 404 returned error can't find the container with id 9b79795217d741601c85e245bcd58b27a8ac2a7bbf517808ee89b067dbf82272 Jul 22 09:40:49.504579 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:49.504539 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" event={"ID":"d9f63887-e905-4e22-8bf9-19a35d61a5e2","Type":"ContainerStarted","Data":"848e6199a4049e03c667bd72214688219f151f2642978aecec56157660de4fac"} Jul 22 09:40:49.504579 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:49.504584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" event={"ID":"d9f63887-e905-4e22-8bf9-19a35d61a5e2","Type":"ContainerStarted","Data":"9b79795217d741601c85e245bcd58b27a8ac2a7bbf517808ee89b067dbf82272"} Jul 22 09:40:49.526269 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:49.526217 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" podStartSLOduration=1.526202535 podStartE2EDuration="1.526202535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:40:49.525199707 +0000 UTC m=+1374.022707144" watchObservedRunningTime="2026-07-22 09:40:49.526202535 +0000 UTC m=+1374.023709971" Jul 22 09:40:51.961535 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:51.961490 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:40:58.572408 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:58.572372 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:58.572850 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:58.572454 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:58.584514 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:58.584490 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:40:59.564119 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:40:59.564092 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:41:09.428409 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:09.428377 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq"] Jul 22 09:41:10.825054 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:10.825018 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz"] Jul 22 09:41:13.347956 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:13.347907 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:41:24.449466 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.449408 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" podUID="a241580e-ba27-4fc1-9e43-59d47b941130" containerName="main" containerID="cri-o://bc9760dc3fac7ffee787b5d4b1ec0b076ab193b391f7d815e01b8713a8a2a695" gracePeriod=45 Jul 22 09:41:24.674337 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.674306 2564 generic.go:356] "Generic (PLEG): container finished" podID="a241580e-ba27-4fc1-9e43-59d47b941130" containerID="bc9760dc3fac7ffee787b5d4b1ec0b076ab193b391f7d815e01b8713a8a2a695" exitCode=0 Jul 22 09:41:24.674492 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.674362 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" event={"ID":"a241580e-ba27-4fc1-9e43-59d47b941130","Type":"ContainerDied","Data":"bc9760dc3fac7ffee787b5d4b1ec0b076ab193b391f7d815e01b8713a8a2a695"} Jul 22 09:41:24.699661 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.699602 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:41:24.797622 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.797591 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-dshm\") pod \"a241580e-ba27-4fc1-9e43-59d47b941130\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " Jul 22 09:41:24.797778 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.797650 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-tmp-dir\") pod \"a241580e-ba27-4fc1-9e43-59d47b941130\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " Jul 22 09:41:24.797823 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.797775 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-model-cache\") pod \"a241580e-ba27-4fc1-9e43-59d47b941130\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " Jul 22 09:41:24.797875 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.797822 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a241580e-ba27-4fc1-9e43-59d47b941130-tls-certs\") pod \"a241580e-ba27-4fc1-9e43-59d47b941130\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " Jul 22 09:41:24.797875 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.797838 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a241580e-ba27-4fc1-9e43-59d47b941130" (UID: "a241580e-ba27-4fc1-9e43-59d47b941130"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:41:24.797875 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.797851 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-home\") pod \"a241580e-ba27-4fc1-9e43-59d47b941130\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " Jul 22 09:41:24.798027 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.797925 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fctz\" (UniqueName: \"kubernetes.io/projected/a241580e-ba27-4fc1-9e43-59d47b941130-kube-api-access-7fctz\") pod \"a241580e-ba27-4fc1-9e43-59d47b941130\" (UID: \"a241580e-ba27-4fc1-9e43-59d47b941130\") " Jul 22 09:41:24.798084 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.798060 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-model-cache" (OuterVolumeSpecName: "model-cache") pod "a241580e-ba27-4fc1-9e43-59d47b941130" (UID: "a241580e-ba27-4fc1-9e43-59d47b941130"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:41:24.798136 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.798071 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-home" (OuterVolumeSpecName: "home") pod "a241580e-ba27-4fc1-9e43-59d47b941130" (UID: "a241580e-ba27-4fc1-9e43-59d47b941130"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:41:24.798286 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.798267 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-tmp-dir\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:41:24.798381 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.798288 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-model-cache\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:41:24.798381 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.798297 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-home\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:41:24.799779 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.799754 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-dshm" (OuterVolumeSpecName: "dshm") pod "a241580e-ba27-4fc1-9e43-59d47b941130" (UID: "a241580e-ba27-4fc1-9e43-59d47b941130"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:41:24.799893 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.799823 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a241580e-ba27-4fc1-9e43-59d47b941130-kube-api-access-7fctz" (OuterVolumeSpecName: "kube-api-access-7fctz") pod "a241580e-ba27-4fc1-9e43-59d47b941130" (UID: "a241580e-ba27-4fc1-9e43-59d47b941130"). InnerVolumeSpecName "kube-api-access-7fctz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:41:24.799985 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.799893 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a241580e-ba27-4fc1-9e43-59d47b941130-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a241580e-ba27-4fc1-9e43-59d47b941130" (UID: "a241580e-ba27-4fc1-9e43-59d47b941130"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:41:24.899748 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.899720 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a241580e-ba27-4fc1-9e43-59d47b941130-dshm\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:41:24.899748 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.899742 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a241580e-ba27-4fc1-9e43-59d47b941130-tls-certs\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:41:24.899913 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:24.899756 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fctz\" (UniqueName: \"kubernetes.io/projected/a241580e-ba27-4fc1-9e43-59d47b941130-kube-api-access-7fctz\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:41:25.681069 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:25.681028 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" event={"ID":"a241580e-ba27-4fc1-9e43-59d47b941130","Type":"ContainerDied","Data":"31d34c181925956b988e378911f9b0c80e2a4ca0d7b0249f827596ce588e9037"} Jul 22 09:41:25.681498 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:25.681083 2564 scope.go:117] "RemoveContainer" containerID="bc9760dc3fac7ffee787b5d4b1ec0b076ab193b391f7d815e01b8713a8a2a695" Jul 22 09:41:25.681498 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:25.681038 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq" Jul 22 09:41:25.703146 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:25.703122 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq"] Jul 22 09:41:25.710964 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:25.710925 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0fe1d095-kserve-748964df59-lhpdq"] Jul 22 09:41:25.844653 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:25.844595 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" podUID="d9f63887-e905-4e22-8bf9-19a35d61a5e2" containerName="main" containerID="cri-o://848e6199a4049e03c667bd72214688219f151f2642978aecec56157660de4fac" gracePeriod=45 Jul 22 09:41:26.016789 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.016764 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a241580e-ba27-4fc1-9e43-59d47b941130" path="/var/lib/kubelet/pods/a241580e-ba27-4fc1-9e43-59d47b941130/volumes" Jul 22 09:41:26.093978 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.093956 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:41:26.211614 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.211542 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zhwf\" (UniqueName: \"kubernetes.io/projected/d9f63887-e905-4e22-8bf9-19a35d61a5e2-kube-api-access-6zhwf\") pod \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " Jul 22 09:41:26.211614 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.211580 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-dshm\") pod \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " Jul 22 09:41:26.211614 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.211602 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-home\") pod \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " Jul 22 09:41:26.211895 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.211635 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9f63887-e905-4e22-8bf9-19a35d61a5e2-tls-certs\") pod \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " Jul 22 09:41:26.211895 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.211674 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-model-cache\") pod \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " Jul 22 09:41:26.211895 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.211729 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-tmp-dir\") pod \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\" (UID: \"d9f63887-e905-4e22-8bf9-19a35d61a5e2\") " Jul 22 09:41:26.211895 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.211857 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-home" (OuterVolumeSpecName: "home") pod "d9f63887-e905-4e22-8bf9-19a35d61a5e2" (UID: "d9f63887-e905-4e22-8bf9-19a35d61a5e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:41:26.212123 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.211961 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9f63887-e905-4e22-8bf9-19a35d61a5e2" (UID: "d9f63887-e905-4e22-8bf9-19a35d61a5e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:41:26.212123 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.212036 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9f63887-e905-4e22-8bf9-19a35d61a5e2" (UID: "d9f63887-e905-4e22-8bf9-19a35d61a5e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:41:26.212123 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.212071 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-home\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:41:26.212123 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.212091 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-model-cache\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:41:26.213709 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.213688 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9f63887-e905-4e22-8bf9-19a35d61a5e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9f63887-e905-4e22-8bf9-19a35d61a5e2" (UID: "d9f63887-e905-4e22-8bf9-19a35d61a5e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:41:26.214101 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.214080 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f63887-e905-4e22-8bf9-19a35d61a5e2-kube-api-access-6zhwf" (OuterVolumeSpecName: "kube-api-access-6zhwf") pod "d9f63887-e905-4e22-8bf9-19a35d61a5e2" (UID: "d9f63887-e905-4e22-8bf9-19a35d61a5e2"). InnerVolumeSpecName "kube-api-access-6zhwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:41:26.214201 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.214080 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-dshm" (OuterVolumeSpecName: "dshm") pod "d9f63887-e905-4e22-8bf9-19a35d61a5e2" (UID: "d9f63887-e905-4e22-8bf9-19a35d61a5e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:41:26.313315 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.313288 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-tmp-dir\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:41:26.313315 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.313311 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zhwf\" (UniqueName: \"kubernetes.io/projected/d9f63887-e905-4e22-8bf9-19a35d61a5e2-kube-api-access-6zhwf\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:41:26.313315 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.313321 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9f63887-e905-4e22-8bf9-19a35d61a5e2-dshm\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:41:26.313537 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.313329 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9f63887-e905-4e22-8bf9-19a35d61a5e2-tls-certs\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:41:26.688140 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.688112 2564 generic.go:356] "Generic (PLEG): container finished" podID="d9f63887-e905-4e22-8bf9-19a35d61a5e2" containerID="848e6199a4049e03c667bd72214688219f151f2642978aecec56157660de4fac" exitCode=0 Jul 22 09:41:26.688580 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.688156 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" event={"ID":"d9f63887-e905-4e22-8bf9-19a35d61a5e2","Type":"ContainerDied","Data":"848e6199a4049e03c667bd72214688219f151f2642978aecec56157660de4fac"} Jul 22 09:41:26.688580 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.688164 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" Jul 22 09:41:26.688580 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.688185 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz" event={"ID":"d9f63887-e905-4e22-8bf9-19a35d61a5e2","Type":"ContainerDied","Data":"9b79795217d741601c85e245bcd58b27a8ac2a7bbf517808ee89b067dbf82272"} Jul 22 09:41:26.688580 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.688201 2564 scope.go:117] "RemoveContainer" containerID="848e6199a4049e03c667bd72214688219f151f2642978aecec56157660de4fac" Jul 22 09:41:26.696831 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.696818 2564 scope.go:117] "RemoveContainer" containerID="848e6199a4049e03c667bd72214688219f151f2642978aecec56157660de4fac" Jul 22 09:41:26.697116 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:41:26.697100 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848e6199a4049e03c667bd72214688219f151f2642978aecec56157660de4fac\": container with ID starting with 848e6199a4049e03c667bd72214688219f151f2642978aecec56157660de4fac not found: ID does not exist" containerID="848e6199a4049e03c667bd72214688219f151f2642978aecec56157660de4fac" Jul 22 09:41:26.697184 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.697123 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848e6199a4049e03c667bd72214688219f151f2642978aecec56157660de4fac"} err="failed to get container status \"848e6199a4049e03c667bd72214688219f151f2642978aecec56157660de4fac\": rpc error: code = NotFound desc = could not find container \"848e6199a4049e03c667bd72214688219f151f2642978aecec56157660de4fac\": container with ID starting with 848e6199a4049e03c667bd72214688219f151f2642978aecec56157660de4fac not found: ID does not exist" Jul 22 09:41:26.709575 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.709552 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz"] Jul 22 09:41:26.715458 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.715433 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0fe1d095-kserve-79b6f67f6c-lkpnz"] Jul 22 09:41:26.850253 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:26.850220 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:41:28.015914 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:28.015876 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9f63887-e905-4e22-8bf9-19a35d61a5e2" path="/var/lib/kubelet/pods/d9f63887-e905-4e22-8bf9-19a35d61a5e2/volumes" Jul 22 09:41:33.643474 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:33.643444 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:41:50.356673 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:41:50.356602 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:42:01.559131 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:42:01.559097 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:42:10.763681 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:42:10.763645 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:42:20.957754 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:42:20.957721 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:42:30.612286 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:42:30.612246 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:42:40.248879 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:42:40.248846 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:42:53.752399 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:42:53.752361 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:42:55.998964 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:42:55.998918 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:42:56.001054 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:42:56.001037 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:43:02.873185 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:43:02.873148 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:43:12.587471 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:43:12.587436 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:43:24.165055 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:43:24.164980 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:43:37.273245 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:43:37.273210 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:43:45.085720 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:43:45.085686 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:43:57.866396 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:43:57.866357 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:44:11.260109 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:44:11.260073 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:44:20.449514 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:44:20.449474 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:44:33.550509 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:44:33.550473 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:44:47.756155 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:44:47.756068 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:45:00.129930 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:00.129893 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d701482-29745225-kcw7s"] Jul 22 09:45:00.130393 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:00.130289 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a241580e-ba27-4fc1-9e43-59d47b941130" containerName="main" Jul 22 09:45:00.130393 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:00.130301 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a241580e-ba27-4fc1-9e43-59d47b941130" containerName="main" Jul 22 09:45:00.130393 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:00.130319 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9f63887-e905-4e22-8bf9-19a35d61a5e2" containerName="main" Jul 22 09:45:00.130393 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:00.130324 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9f63887-e905-4e22-8bf9-19a35d61a5e2" containerName="main" Jul 22 09:45:00.130393 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:00.130376 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9f63887-e905-4e22-8bf9-19a35d61a5e2" containerName="main" Jul 22 09:45:00.130393 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:00.130386 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a241580e-ba27-4fc1-9e43-59d47b941130" containerName="main" Jul 22 09:45:00.133388 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:00.133372 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d701482-29745225-kcw7s" Jul 22 09:45:00.136035 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:00.136015 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 09:45:00.143252 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:00.143227 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d701482-29745225-kcw7s"] Jul 22 09:45:00.144624 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:00.144606 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d701482-29745225-kcw7s" Jul 22 09:45:00.274309 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:00.274283 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d701482-29745225-kcw7s"] Jul 22 09:45:00.276759 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:45:00.276728 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3bff16_d76d_4b52_a738_4b1a19944f2f.slice/crio-482816df9a64580c3708c85de27bf45f2a27e1b8cfc1a975d793ca751e357d39 WatchSource:0}: Error finding container 482816df9a64580c3708c85de27bf45f2a27e1b8cfc1a975d793ca751e357d39: Status 404 returned error can't find the container with id 482816df9a64580c3708c85de27bf45f2a27e1b8cfc1a975d793ca751e357d39 Jul 22 09:45:00.279082 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:00.279065 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 09:45:00.711425 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:00.711389 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d701482-29745225-kcw7s" event={"ID":"9f3bff16-d76d-4b52-a738-4b1a19944f2f","Type":"ContainerStarted","Data":"482816df9a64580c3708c85de27bf45f2a27e1b8cfc1a975d793ca751e357d39"} Jul 22 09:45:02.722457 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:02.722413 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d701482-29745225-kcw7s" event={"ID":"9f3bff16-d76d-4b52-a738-4b1a19944f2f","Type":"ContainerStarted","Data":"074d2d93af0aaf14ab0f214bad4cf563bccf2f419bf3af8f971bf5b28f15e17f"} Jul 22 09:45:02.738715 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:02.738666 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d701482-29745225-kcw7s" podStartSLOduration=0.636711441 podStartE2EDuration="2.738649537s" totalImagesPullingTime="2.101938096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:45:02.738035791 +0000 UTC m=+1627.235543230" watchObservedRunningTime="2026-07-22 09:45:02.738649537 +0000 UTC m=+1627.236156974" Jul 22 09:45:03.729192 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:03.729159 2564 generic.go:356] "Generic (PLEG): container finished" podID="9f3bff16-d76d-4b52-a738-4b1a19944f2f" containerID="074d2d93af0aaf14ab0f214bad4cf563bccf2f419bf3af8f971bf5b28f15e17f" exitCode=7 Jul 22 09:45:03.729602 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:03.729225 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d701482-29745225-kcw7s" event={"ID":"9f3bff16-d76d-4b52-a738-4b1a19944f2f","Type":"ContainerDied","Data":"074d2d93af0aaf14ab0f214bad4cf563bccf2f419bf3af8f971bf5b28f15e17f"} Jul 22 09:45:03.729602 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:03.729371 2564 scope.go:117] "RemoveContainer" containerID="074d2d93af0aaf14ab0f214bad4cf563bccf2f419bf3af8f971bf5b28f15e17f" Jul 22 09:45:03.857001 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:03.856969 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:45:04.735881 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:04.735834 2564 generic.go:356] "Generic (PLEG): container finished" podID="9f3bff16-d76d-4b52-a738-4b1a19944f2f" containerID="88039be9f0c47c03dca1955cc024d12ce3c5ec05f7aeca6028e7eb48de539814" exitCode=0 Jul 22 09:45:04.736356 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:04.735906 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d701482-29745225-kcw7s" event={"ID":"9f3bff16-d76d-4b52-a738-4b1a19944f2f","Type":"ContainerDied","Data":"88039be9f0c47c03dca1955cc024d12ce3c5ec05f7aeca6028e7eb48de539814"} Jul 22 09:45:04.736356 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:04.735987 2564 scope.go:117] "RemoveContainer" containerID="074d2d93af0aaf14ab0f214bad4cf563bccf2f419bf3af8f971bf5b28f15e17f" Jul 22 09:45:05.869498 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:05.869474 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d701482-29745225-kcw7s" Jul 22 09:45:06.748324 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:06.748294 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d701482-29745225-kcw7s" Jul 22 09:45:06.748324 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:06.748307 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d701482-29745225-kcw7s" event={"ID":"9f3bff16-d76d-4b52-a738-4b1a19944f2f","Type":"ContainerDied","Data":"482816df9a64580c3708c85de27bf45f2a27e1b8cfc1a975d793ca751e357d39"} Jul 22 09:45:06.748544 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:06.748350 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="482816df9a64580c3708c85de27bf45f2a27e1b8cfc1a975d793ca751e357d39" Jul 22 09:45:32.153189 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:32.153154 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d701482-29745225-kcw7s"] Jul 22 09:45:32.155858 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:32.155834 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1d701482-29745225-kcw7s"] Jul 22 09:45:34.014906 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:34.014873 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f3bff16-d76d-4b52-a738-4b1a19944f2f" path="/var/lib/kubelet/pods/9f3bff16-d76d-4b52-a738-4b1a19944f2f/volumes" Jul 22 09:45:35.045975 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:35.045929 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:45:47.353572 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:45:47.353529 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:46:05.472484 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:05.472448 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6"] Jul 22 09:46:05.472863 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:05.472820 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f3bff16-d76d-4b52-a738-4b1a19944f2f" containerName="cleanup" Jul 22 09:46:05.472863 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:05.472831 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f3bff16-d76d-4b52-a738-4b1a19944f2f" containerName="cleanup" Jul 22 09:46:05.472863 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:05.472846 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f3bff16-d76d-4b52-a738-4b1a19944f2f" containerName="cleanup" Jul 22 09:46:05.472863 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:05.472852 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f3bff16-d76d-4b52-a738-4b1a19944f2f" containerName="cleanup" Jul 22 09:46:05.473044 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:05.472945 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f3bff16-d76d-4b52-a738-4b1a19944f2f" containerName="cleanup" Jul 22 09:46:05.476128 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:05.476108 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6" Jul 22 09:46:05.479477 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:05.479450 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 09:46:05.479572 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:05.479510 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 09:46:05.487657 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:05.487632 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6"] Jul 22 09:46:05.573417 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:05.573379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4134f3e6-588b-45c0-993b-32944034f88d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6\" (UID: \"4134f3e6-588b-45c0-993b-32944034f88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6" Jul 22 09:46:05.674742 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:05.674703 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4134f3e6-588b-45c0-993b-32944034f88d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6\" (UID: \"4134f3e6-588b-45c0-993b-32944034f88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6" Jul 22 09:46:05.675359 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:05.675340 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4134f3e6-588b-45c0-993b-32944034f88d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6\" (UID: \"4134f3e6-588b-45c0-993b-32944034f88d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6" Jul 22 09:46:05.788044 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:05.787961 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6" Jul 22 09:46:05.919326 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:05.919303 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6"] Jul 22 09:46:05.922214 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:46:05.922186 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4134f3e6_588b_45c0_993b_32944034f88d.slice/crio-c6857975d653bc4b106d828d1d97eaea9521add45d60b9f22cbd23c516bfd2b7 WatchSource:0}: Error finding container c6857975d653bc4b106d828d1d97eaea9521add45d60b9f22cbd23c516bfd2b7: Status 404 returned error can't find the container with id c6857975d653bc4b106d828d1d97eaea9521add45d60b9f22cbd23c516bfd2b7 Jul 22 09:46:06.035298 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:06.035266 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6" event={"ID":"4134f3e6-588b-45c0-993b-32944034f88d","Type":"ContainerStarted","Data":"c6857975d653bc4b106d828d1d97eaea9521add45d60b9f22cbd23c516bfd2b7"} Jul 22 09:46:07.042649 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:07.042615 2564 generic.go:356] "Generic (PLEG): container finished" podID="4134f3e6-588b-45c0-993b-32944034f88d" containerID="f6148fd33b83a8fb3ab05729cdc905339752b62fb9da937dbd1f212f5b0e569b" exitCode=0 Jul 22 09:46:07.043037 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:07.042700 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6" event={"ID":"4134f3e6-588b-45c0-993b-32944034f88d","Type":"ContainerDied","Data":"f6148fd33b83a8fb3ab05729cdc905339752b62fb9da937dbd1f212f5b0e569b"} Jul 22 09:46:08.171429 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:08.171407 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6" Jul 22 09:46:08.299275 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:08.299194 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4134f3e6-588b-45c0-993b-32944034f88d-maas-api-service-ca\") pod \"4134f3e6-588b-45c0-993b-32944034f88d\" (UID: \"4134f3e6-588b-45c0-993b-32944034f88d\") " Jul 22 09:46:08.299523 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:08.299500 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4134f3e6-588b-45c0-993b-32944034f88d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4134f3e6-588b-45c0-993b-32944034f88d" (UID: "4134f3e6-588b-45c0-993b-32944034f88d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:46:08.400137 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:08.400108 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4134f3e6-588b-45c0-993b-32944034f88d-maas-api-service-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:46:09.053686 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:09.053651 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6" event={"ID":"4134f3e6-588b-45c0-993b-32944034f88d","Type":"ContainerDied","Data":"c6857975d653bc4b106d828d1d97eaea9521add45d60b9f22cbd23c516bfd2b7"} Jul 22 09:46:09.053882 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:09.053703 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6857975d653bc4b106d828d1d97eaea9521add45d60b9f22cbd23c516bfd2b7" Jul 22 09:46:09.053882 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:09.053661 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6" Jul 22 09:46:18.856489 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:18.856411 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:46:20.557527 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:20.557482 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6"] Jul 22 09:46:20.560789 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:20.560764 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9e388426-4c6n6"] Jul 22 09:46:22.015201 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:22.015171 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4134f3e6-588b-45c0-993b-32944034f88d" path="/var/lib/kubelet/pods/4134f3e6-588b-45c0-993b-32944034f88d/volumes" Jul 22 09:46:30.549150 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:30.549115 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:46:49.413447 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:49.413415 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-424990ec-n785l"] Jul 22 09:46:49.413878 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:49.413807 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4134f3e6-588b-45c0-993b-32944034f88d" containerName="revoke-keys" Jul 22 09:46:49.413878 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:49.413818 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4134f3e6-588b-45c0-993b-32944034f88d" containerName="revoke-keys" Jul 22 09:46:49.413993 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:49.413884 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f3bff16-d76d-4b52-a738-4b1a19944f2f" containerName="cleanup" Jul 22 09:46:49.413993 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:49.413893 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4134f3e6-588b-45c0-993b-32944034f88d" containerName="revoke-keys" Jul 22 09:46:49.416988 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:49.416972 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-424990ec-n785l" Jul 22 09:46:49.420174 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:49.420152 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 09:46:49.420284 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:49.420189 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 09:46:49.430845 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:49.430825 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-424990ec-n785l"] Jul 22 09:46:49.469906 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:49.469878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bc5a04-83db-444c-bd90-29183200c9f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-424990ec-n785l\" (UID: \"09bc5a04-83db-444c-bd90-29183200c9f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-424990ec-n785l" Jul 22 09:46:49.571362 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:49.571322 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bc5a04-83db-444c-bd90-29183200c9f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-424990ec-n785l\" (UID: \"09bc5a04-83db-444c-bd90-29183200c9f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-424990ec-n785l" Jul 22 09:46:49.572024 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:49.572005 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bc5a04-83db-444c-bd90-29183200c9f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-424990ec-n785l\" (UID: \"09bc5a04-83db-444c-bd90-29183200c9f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-424990ec-n785l" Jul 22 09:46:49.727221 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:49.727188 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-424990ec-n785l" Jul 22 09:46:50.071514 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:50.071492 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-424990ec-n785l"] Jul 22 09:46:50.073621 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:46:50.073597 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bc5a04_83db_444c_bd90_29183200c9f5.slice/crio-f8c2e64fcc32c5566f4594d291c35219442affa0837233531cc9864e33c71e94 WatchSource:0}: Error finding container f8c2e64fcc32c5566f4594d291c35219442affa0837233531cc9864e33c71e94: Status 404 returned error can't find the container with id f8c2e64fcc32c5566f4594d291c35219442affa0837233531cc9864e33c71e94 Jul 22 09:46:50.250679 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:50.250645 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-424990ec-n785l" event={"ID":"09bc5a04-83db-444c-bd90-29183200c9f5","Type":"ContainerStarted","Data":"f8c2e64fcc32c5566f4594d291c35219442affa0837233531cc9864e33c71e94"} Jul 22 09:46:51.256583 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:51.256550 2564 generic.go:356] "Generic (PLEG): container finished" podID="09bc5a04-83db-444c-bd90-29183200c9f5" containerID="3cd4f3c21211a3fe2fb6d5057d6a15a1f4263bcad3a82917c1f02cd608311a2d" exitCode=0 Jul 22 09:46:51.256583 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:51.256587 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-424990ec-n785l" event={"ID":"09bc5a04-83db-444c-bd90-29183200c9f5","Type":"ContainerDied","Data":"3cd4f3c21211a3fe2fb6d5057d6a15a1f4263bcad3a82917c1f02cd608311a2d"} Jul 22 09:46:52.390951 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:52.390910 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-424990ec-n785l" Jul 22 09:46:52.497857 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:52.497829 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bc5a04-83db-444c-bd90-29183200c9f5-maas-api-service-ca\") pod \"09bc5a04-83db-444c-bd90-29183200c9f5\" (UID: \"09bc5a04-83db-444c-bd90-29183200c9f5\") " Jul 22 09:46:52.498170 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:52.498149 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09bc5a04-83db-444c-bd90-29183200c9f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09bc5a04-83db-444c-bd90-29183200c9f5" (UID: "09bc5a04-83db-444c-bd90-29183200c9f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:46:52.599127 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:52.599056 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bc5a04-83db-444c-bd90-29183200c9f5-maas-api-service-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:46:53.266857 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:53.266822 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-424990ec-n785l" event={"ID":"09bc5a04-83db-444c-bd90-29183200c9f5","Type":"ContainerDied","Data":"f8c2e64fcc32c5566f4594d291c35219442affa0837233531cc9864e33c71e94"} Jul 22 09:46:53.266857 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:53.266854 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-424990ec-n785l" Jul 22 09:46:53.267081 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:46:53.266871 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8c2e64fcc32c5566f4594d291c35219442affa0837233531cc9864e33c71e94" Jul 22 09:47:02.850518 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:02.850477 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:47:04.480442 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:04.480404 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-424990ec-n785l"] Jul 22 09:47:04.484572 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:04.484547 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-424990ec-n785l"] Jul 22 09:47:06.014953 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:06.014915 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09bc5a04-83db-444c-bd90-29183200c9f5" path="/var/lib/kubelet/pods/09bc5a04-83db-444c-bd90-29183200c9f5/volumes" Jul 22 09:47:15.745636 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:15.745597 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:47:33.730463 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:33.730426 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh"] Jul 22 09:47:33.730928 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:33.730801 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09bc5a04-83db-444c-bd90-29183200c9f5" containerName="revoke-keys" Jul 22 09:47:33.730928 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:33.730811 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="09bc5a04-83db-444c-bd90-29183200c9f5" containerName="revoke-keys" Jul 22 09:47:33.730928 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:33.730878 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="09bc5a04-83db-444c-bd90-29183200c9f5" containerName="revoke-keys" Jul 22 09:47:33.733975 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:33.733959 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh" Jul 22 09:47:33.737061 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:33.737042 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 09:47:33.737389 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:33.737374 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 09:47:33.747872 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:33.747851 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh"] Jul 22 09:47:33.786102 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:33.786076 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d39a36af-60a9-4d8e-83de-d69df1315cbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d055000b-2kzbh\" (UID: \"d39a36af-60a9-4d8e-83de-d69df1315cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh" Jul 22 09:47:33.887275 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:33.887236 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d39a36af-60a9-4d8e-83de-d69df1315cbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d055000b-2kzbh\" (UID: \"d39a36af-60a9-4d8e-83de-d69df1315cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh" Jul 22 09:47:33.887913 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:33.887891 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d39a36af-60a9-4d8e-83de-d69df1315cbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d055000b-2kzbh\" (UID: \"d39a36af-60a9-4d8e-83de-d69df1315cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh" Jul 22 09:47:34.043395 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:34.043322 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh" Jul 22 09:47:34.176767 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:34.176746 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh"] Jul 22 09:47:34.178636 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:47:34.178608 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39a36af_60a9_4d8e_83de_d69df1315cbf.slice/crio-25713b19516510c7963c300ed3b44e0f51ae3a280e9bf5cbbe32e0b07acc9b85 WatchSource:0}: Error finding container 25713b19516510c7963c300ed3b44e0f51ae3a280e9bf5cbbe32e0b07acc9b85: Status 404 returned error can't find the container with id 25713b19516510c7963c300ed3b44e0f51ae3a280e9bf5cbbe32e0b07acc9b85 Jul 22 09:47:34.464144 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:34.464105 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh" event={"ID":"d39a36af-60a9-4d8e-83de-d69df1315cbf","Type":"ContainerStarted","Data":"47e28eae8748a8fed1ccc45509fefddde7be6e724573083384d7acb5a4ff43f6"} Jul 22 09:47:34.464311 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:34.464165 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh" event={"ID":"d39a36af-60a9-4d8e-83de-d69df1315cbf","Type":"ContainerStarted","Data":"25713b19516510c7963c300ed3b44e0f51ae3a280e9bf5cbbe32e0b07acc9b85"} Jul 22 09:47:34.491692 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:34.491647 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh" podStartSLOduration=1.491633277 podStartE2EDuration="1.491633277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:47:34.490248335 +0000 UTC m=+1778.987755771" watchObservedRunningTime="2026-07-22 09:47:34.491633277 +0000 UTC m=+1778.989140712" Jul 22 09:47:35.469972 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:35.469911 2564 generic.go:356] "Generic (PLEG): container finished" podID="d39a36af-60a9-4d8e-83de-d69df1315cbf" containerID="47e28eae8748a8fed1ccc45509fefddde7be6e724573083384d7acb5a4ff43f6" exitCode=0 Jul 22 09:47:35.470350 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:35.469983 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh" event={"ID":"d39a36af-60a9-4d8e-83de-d69df1315cbf","Type":"ContainerDied","Data":"47e28eae8748a8fed1ccc45509fefddde7be6e724573083384d7acb5a4ff43f6"} Jul 22 09:47:36.600179 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:36.600155 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh" Jul 22 09:47:36.715777 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:36.715745 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d39a36af-60a9-4d8e-83de-d69df1315cbf-maas-api-service-ca\") pod \"d39a36af-60a9-4d8e-83de-d69df1315cbf\" (UID: \"d39a36af-60a9-4d8e-83de-d69df1315cbf\") " Jul 22 09:47:36.716172 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:36.716152 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39a36af-60a9-4d8e-83de-d69df1315cbf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d39a36af-60a9-4d8e-83de-d69df1315cbf" (UID: "d39a36af-60a9-4d8e-83de-d69df1315cbf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:47:36.817127 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:36.817052 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d39a36af-60a9-4d8e-83de-d69df1315cbf-maas-api-service-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:47:37.480827 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:37.480789 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh" event={"ID":"d39a36af-60a9-4d8e-83de-d69df1315cbf","Type":"ContainerDied","Data":"25713b19516510c7963c300ed3b44e0f51ae3a280e9bf5cbbe32e0b07acc9b85"} Jul 22 09:47:37.480827 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:37.480819 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh" Jul 22 09:47:37.480827 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:37.480837 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25713b19516510c7963c300ed3b44e0f51ae3a280e9bf5cbbe32e0b07acc9b85" Jul 22 09:47:46.848620 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:46.848583 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:47:48.804633 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:48.804556 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh"] Jul 22 09:47:48.807005 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:48.806983 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d055000b-2kzbh"] Jul 22 09:47:50.014437 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:50.014401 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d39a36af-60a9-4d8e-83de-d69df1315cbf" path="/var/lib/kubelet/pods/d39a36af-60a9-4d8e-83de-d69df1315cbf/volumes" Jul 22 09:47:56.031055 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:56.031023 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:47:56.034331 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:47:56.034309 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:49:14.744576 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:14.744540 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:49:26.361246 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:26.361163 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:49:40.056083 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:40.056051 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:49:42.360862 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:42.360829 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf"] Jul 22 09:49:42.361260 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:42.361246 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d39a36af-60a9-4d8e-83de-d69df1315cbf" containerName="revoke-keys" Jul 22 09:49:42.361260 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:42.361261 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d39a36af-60a9-4d8e-83de-d69df1315cbf" containerName="revoke-keys" Jul 22 09:49:42.361335 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:42.361313 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d39a36af-60a9-4d8e-83de-d69df1315cbf" containerName="revoke-keys" Jul 22 09:49:42.364483 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:42.364467 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf" Jul 22 09:49:42.372464 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:42.372441 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 09:49:42.372664 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:42.372643 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 09:49:42.381664 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:42.381640 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf"] Jul 22 09:49:42.481092 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:42.481058 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/965f1210-6952-46bc-899d-b7ef6ea17c31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf\" (UID: \"965f1210-6952-46bc-899d-b7ef6ea17c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf" Jul 22 09:49:42.582359 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:42.582326 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/965f1210-6952-46bc-899d-b7ef6ea17c31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf\" (UID: \"965f1210-6952-46bc-899d-b7ef6ea17c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf" Jul 22 09:49:42.583004 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:42.582983 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/965f1210-6952-46bc-899d-b7ef6ea17c31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf\" (UID: \"965f1210-6952-46bc-899d-b7ef6ea17c31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf" Jul 22 09:49:42.675196 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:42.675117 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf" Jul 22 09:49:42.805211 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:42.805125 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf"] Jul 22 09:49:42.807555 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:49:42.807528 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965f1210_6952_46bc_899d_b7ef6ea17c31.slice/crio-6eecf8de90137e13e183c99a231f3032604a47f531978832273589fe2ab63269 WatchSource:0}: Error finding container 6eecf8de90137e13e183c99a231f3032604a47f531978832273589fe2ab63269: Status 404 returned error can't find the container with id 6eecf8de90137e13e183c99a231f3032604a47f531978832273589fe2ab63269 Jul 22 09:49:43.111075 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:43.111040 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf" event={"ID":"965f1210-6952-46bc-899d-b7ef6ea17c31","Type":"ContainerStarted","Data":"6eecf8de90137e13e183c99a231f3032604a47f531978832273589fe2ab63269"} Jul 22 09:49:44.116825 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:44.116789 2564 generic.go:356] "Generic (PLEG): container finished" podID="965f1210-6952-46bc-899d-b7ef6ea17c31" containerID="a8e9fd054dc7214fa7fbc6752f2234d5b76ee0446e177fbe43b9de007d93f43e" exitCode=0 Jul 22 09:49:44.117267 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:44.116835 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf" event={"ID":"965f1210-6952-46bc-899d-b7ef6ea17c31","Type":"ContainerDied","Data":"a8e9fd054dc7214fa7fbc6752f2234d5b76ee0446e177fbe43b9de007d93f43e"} Jul 22 09:49:45.249534 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:45.249509 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf" Jul 22 09:49:45.308453 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:45.308423 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/965f1210-6952-46bc-899d-b7ef6ea17c31-maas-api-service-ca\") pod \"965f1210-6952-46bc-899d-b7ef6ea17c31\" (UID: \"965f1210-6952-46bc-899d-b7ef6ea17c31\") " Jul 22 09:49:45.315012 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:45.309324 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965f1210-6952-46bc-899d-b7ef6ea17c31-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "965f1210-6952-46bc-899d-b7ef6ea17c31" (UID: "965f1210-6952-46bc-899d-b7ef6ea17c31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:49:45.315012 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:45.309675 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/965f1210-6952-46bc-899d-b7ef6ea17c31-maas-api-service-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:49:46.127661 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:46.127631 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf" Jul 22 09:49:46.127661 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:46.127643 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf" event={"ID":"965f1210-6952-46bc-899d-b7ef6ea17c31","Type":"ContainerDied","Data":"6eecf8de90137e13e183c99a231f3032604a47f531978832273589fe2ab63269"} Jul 22 09:49:46.127917 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:46.127696 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eecf8de90137e13e183c99a231f3032604a47f531978832273589fe2ab63269" Jul 22 09:49:55.771071 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:55.771024 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:49:57.453333 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:57.453301 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf"] Jul 22 09:49:57.454878 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:57.454856 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5ba85bad-llmpf"] Jul 22 09:49:58.014885 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:49:58.014850 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="965f1210-6952-46bc-899d-b7ef6ea17c31" path="/var/lib/kubelet/pods/965f1210-6952-46bc-899d-b7ef6ea17c31/volumes" Jul 22 09:50:34.554531 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:34.554488 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:50:40.548160 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.548122 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh"] Jul 22 09:50:40.548520 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.548492 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="965f1210-6952-46bc-899d-b7ef6ea17c31" containerName="revoke-keys" Jul 22 09:50:40.548520 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.548504 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="965f1210-6952-46bc-899d-b7ef6ea17c31" containerName="revoke-keys" Jul 22 09:50:40.548596 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.548579 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="965f1210-6952-46bc-899d-b7ef6ea17c31" containerName="revoke-keys" Jul 22 09:50:40.551493 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.551473 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.555067 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.555045 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6285b203\"/\"default-dockercfg-mhs99\"" Jul 22 09:50:40.555276 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.555047 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6285b203\"/\"e2e-lifecycle-model-6285b203-kserve-self-signed-certs\"" Jul 22 09:50:40.555374 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.555047 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6285b203\"/\"kube-root-ca.crt\"" Jul 22 09:50:40.555374 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.555072 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6285b203\"/\"openshift-service-ca.crt\"" Jul 22 09:50:40.562195 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.562174 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh"] Jul 22 09:50:40.622435 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.622403 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-tmp-dir\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.622601 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.622453 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9wg5\" (UniqueName: \"kubernetes.io/projected/613a2a26-4950-46f4-8a31-4068619076e4-kube-api-access-l9wg5\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.622601 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.622474 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-dshm\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.622601 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.622544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/613a2a26-4950-46f4-8a31-4068619076e4-tls-certs\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.622601 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.622581 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-model-cache\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.622742 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.622637 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-home\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.723517 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.723478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9wg5\" (UniqueName: \"kubernetes.io/projected/613a2a26-4950-46f4-8a31-4068619076e4-kube-api-access-l9wg5\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.723517 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.723515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-dshm\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.723737 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.723642 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/613a2a26-4950-46f4-8a31-4068619076e4-tls-certs\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.723737 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.723693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-model-cache\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.723814 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.723756 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-home\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.723814 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.723784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-tmp-dir\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.724097 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.724076 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-model-cache\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.724190 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.724113 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-tmp-dir\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.724190 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.724147 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-home\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.725717 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.725697 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-dshm\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.726018 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.725997 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/613a2a26-4950-46f4-8a31-4068619076e4-tls-certs\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.732009 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.731990 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9wg5\" (UniqueName: \"kubernetes.io/projected/613a2a26-4950-46f4-8a31-4068619076e4-kube-api-access-l9wg5\") pod \"e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.863636 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.863547 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:40.998426 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:40.998401 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh"] Jul 22 09:50:41.000547 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:50:41.000520 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod613a2a26_4950_46f4_8a31_4068619076e4.slice/crio-b973a7e4d3da6bb16e307fa038fcfd61710847ec30951ad20a9caed030639c17 WatchSource:0}: Error finding container b973a7e4d3da6bb16e307fa038fcfd61710847ec30951ad20a9caed030639c17: Status 404 returned error can't find the container with id b973a7e4d3da6bb16e307fa038fcfd61710847ec30951ad20a9caed030639c17 Jul 22 09:50:41.401697 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:41.401659 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" event={"ID":"613a2a26-4950-46f4-8a31-4068619076e4","Type":"ContainerStarted","Data":"aa9c3502870f6b0ef430a7a6f711a08fc0e90c83b8b8e41e72e9fd3cf630dc9c"} Jul 22 09:50:41.401697 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:41.401701 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" event={"ID":"613a2a26-4950-46f4-8a31-4068619076e4","Type":"ContainerStarted","Data":"b973a7e4d3da6bb16e307fa038fcfd61710847ec30951ad20a9caed030639c17"} Jul 22 09:50:41.420563 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:41.420509 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" podStartSLOduration=1.4204938120000001 podStartE2EDuration="1.420493812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:50:41.419721053 +0000 UTC m=+1965.917228494" watchObservedRunningTime="2026-07-22 09:50:41.420493812 +0000 UTC m=+1965.918001247" Jul 22 09:50:46.057755 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:46.057714 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:50:50.864183 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:50.864087 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:50.864183 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:50.864130 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:50.876919 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:50.876893 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:51.463289 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:51.463260 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:50:57.854683 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:50:57.854645 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:51:19.474875 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:19.474835 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:51:30.570403 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:30.570371 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:51:32.638723 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:32.638690 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh"] Jul 22 09:51:47.658633 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:47.658574 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" podUID="613a2a26-4950-46f4-8a31-4068619076e4" containerName="main" containerID="cri-o://aa9c3502870f6b0ef430a7a6f711a08fc0e90c83b8b8e41e72e9fd3cf630dc9c" gracePeriod=45 Jul 22 09:51:47.903553 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:47.903530 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:51:48.034987 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.034956 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-home\") pod \"613a2a26-4950-46f4-8a31-4068619076e4\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " Jul 22 09:51:48.035174 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.035017 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9wg5\" (UniqueName: \"kubernetes.io/projected/613a2a26-4950-46f4-8a31-4068619076e4-kube-api-access-l9wg5\") pod \"613a2a26-4950-46f4-8a31-4068619076e4\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " Jul 22 09:51:48.035174 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.035070 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-tmp-dir\") pod \"613a2a26-4950-46f4-8a31-4068619076e4\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " Jul 22 09:51:48.035174 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.035109 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-model-cache\") pod \"613a2a26-4950-46f4-8a31-4068619076e4\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " Jul 22 09:51:48.035174 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.035143 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-dshm\") pod \"613a2a26-4950-46f4-8a31-4068619076e4\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " Jul 22 09:51:48.035751 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.035193 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/613a2a26-4950-46f4-8a31-4068619076e4-tls-certs\") pod \"613a2a26-4950-46f4-8a31-4068619076e4\" (UID: \"613a2a26-4950-46f4-8a31-4068619076e4\") " Jul 22 09:51:48.035751 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.035206 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-home" (OuterVolumeSpecName: "home") pod "613a2a26-4950-46f4-8a31-4068619076e4" (UID: "613a2a26-4950-46f4-8a31-4068619076e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:51:48.035751 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.035364 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "613a2a26-4950-46f4-8a31-4068619076e4" (UID: "613a2a26-4950-46f4-8a31-4068619076e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:51:48.035751 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.035376 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "613a2a26-4950-46f4-8a31-4068619076e4" (UID: "613a2a26-4950-46f4-8a31-4068619076e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:51:48.035751 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.035465 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-home\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:51:48.035751 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.035476 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-tmp-dir\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:51:48.035751 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.035486 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-model-cache\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:51:48.037284 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.037260 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/613a2a26-4950-46f4-8a31-4068619076e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "613a2a26-4950-46f4-8a31-4068619076e4" (UID: "613a2a26-4950-46f4-8a31-4068619076e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:51:48.037378 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.037287 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-dshm" (OuterVolumeSpecName: "dshm") pod "613a2a26-4950-46f4-8a31-4068619076e4" (UID: "613a2a26-4950-46f4-8a31-4068619076e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:51:48.037378 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.037339 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613a2a26-4950-46f4-8a31-4068619076e4-kube-api-access-l9wg5" (OuterVolumeSpecName: "kube-api-access-l9wg5") pod "613a2a26-4950-46f4-8a31-4068619076e4" (UID: "613a2a26-4950-46f4-8a31-4068619076e4"). InnerVolumeSpecName "kube-api-access-l9wg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:51:48.136311 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.136277 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/613a2a26-4950-46f4-8a31-4068619076e4-dshm\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:51:48.136311 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.136303 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/613a2a26-4950-46f4-8a31-4068619076e4-tls-certs\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:51:48.136311 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.136312 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9wg5\" (UniqueName: \"kubernetes.io/projected/613a2a26-4950-46f4-8a31-4068619076e4-kube-api-access-l9wg5\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:51:48.742563 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.742530 2564 generic.go:356] "Generic (PLEG): container finished" podID="613a2a26-4950-46f4-8a31-4068619076e4" containerID="aa9c3502870f6b0ef430a7a6f711a08fc0e90c83b8b8e41e72e9fd3cf630dc9c" exitCode=0 Jul 22 09:51:48.743001 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.742570 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" Jul 22 09:51:48.743001 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.742603 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" event={"ID":"613a2a26-4950-46f4-8a31-4068619076e4","Type":"ContainerDied","Data":"aa9c3502870f6b0ef430a7a6f711a08fc0e90c83b8b8e41e72e9fd3cf630dc9c"} Jul 22 09:51:48.743001 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.742635 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh" event={"ID":"613a2a26-4950-46f4-8a31-4068619076e4","Type":"ContainerDied","Data":"b973a7e4d3da6bb16e307fa038fcfd61710847ec30951ad20a9caed030639c17"} Jul 22 09:51:48.743001 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.742651 2564 scope.go:117] "RemoveContainer" containerID="aa9c3502870f6b0ef430a7a6f711a08fc0e90c83b8b8e41e72e9fd3cf630dc9c" Jul 22 09:51:48.751434 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.751418 2564 scope.go:117] "RemoveContainer" containerID="aa9c3502870f6b0ef430a7a6f711a08fc0e90c83b8b8e41e72e9fd3cf630dc9c" Jul 22 09:51:48.751671 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:51:48.751654 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9c3502870f6b0ef430a7a6f711a08fc0e90c83b8b8e41e72e9fd3cf630dc9c\": container with ID starting with aa9c3502870f6b0ef430a7a6f711a08fc0e90c83b8b8e41e72e9fd3cf630dc9c not found: ID does not exist" containerID="aa9c3502870f6b0ef430a7a6f711a08fc0e90c83b8b8e41e72e9fd3cf630dc9c" Jul 22 09:51:48.751718 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.751680 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9c3502870f6b0ef430a7a6f711a08fc0e90c83b8b8e41e72e9fd3cf630dc9c"} err="failed to get container status \"aa9c3502870f6b0ef430a7a6f711a08fc0e90c83b8b8e41e72e9fd3cf630dc9c\": rpc error: code = NotFound desc = could not find container \"aa9c3502870f6b0ef430a7a6f711a08fc0e90c83b8b8e41e72e9fd3cf630dc9c\": container with ID starting with aa9c3502870f6b0ef430a7a6f711a08fc0e90c83b8b8e41e72e9fd3cf630dc9c not found: ID does not exist" Jul 22 09:51:48.781367 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.781345 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh"] Jul 22 09:51:48.794903 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:48.794883 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6285b203/e2e-lifecycle-model-6285b203-kserve-8687566667-sk5xh"] Jul 22 09:51:50.014661 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:50.014624 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="613a2a26-4950-46f4-8a31-4068619076e4" path="/var/lib/kubelet/pods/613a2a26-4950-46f4-8a31-4068619076e4/volumes" Jul 22 09:51:56.131884 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:51:56.131851 2564 scope.go:117] "RemoveContainer" containerID="88039be9f0c47c03dca1955cc024d12ce3c5ec05f7aeca6028e7eb48de539814" Jul 22 09:52:28.407136 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:52:28.407058 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:52:36.972921 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:52:36.972885 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:52:56.061719 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:52:56.061689 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:52:56.067540 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:52:56.067520 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:52:56.155117 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:52:56.155094 2564 scope.go:117] "RemoveContainer" containerID="f6148fd33b83a8fb3ab05729cdc905339752b62fb9da937dbd1f212f5b0e569b" Jul 22 09:52:56.163076 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:52:56.163055 2564 scope.go:117] "RemoveContainer" containerID="3cd4f3c21211a3fe2fb6d5057d6a15a1f4263bcad3a82917c1f02cd608311a2d" Jul 22 09:53:23.897326 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:23.897286 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:53:36.564135 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:36.564101 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:53:40.464209 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.464178 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb"] Jul 22 09:53:40.464572 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.464560 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="613a2a26-4950-46f4-8a31-4068619076e4" containerName="main" Jul 22 09:53:40.464572 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.464573 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="613a2a26-4950-46f4-8a31-4068619076e4" containerName="main" Jul 22 09:53:40.464647 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.464639 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="613a2a26-4950-46f4-8a31-4068619076e4" containerName="main" Jul 22 09:53:40.467594 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.467577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.470215 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.470192 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-982fc1\"/\"openshift-service-ca.crt\"" Jul 22 09:53:40.471161 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.471146 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-982fc1\"/\"kube-root-ca.crt\"" Jul 22 09:53:40.471266 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.471245 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-982fc1\"/\"default-dockercfg-dm85f\"" Jul 22 09:53:40.471331 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.471251 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-982fc1\"/\"test-model-982fc1-kserve-self-signed-certs\"" Jul 22 09:53:40.478968 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.478921 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb"] Jul 22 09:53:40.573286 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.573254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-model-cache\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.573445 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.573297 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh9kl\" (UniqueName: \"kubernetes.io/projected/edc58a75-4395-44a0-bd99-51a126a3f929-kube-api-access-nh9kl\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.573445 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.573322 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-tmp-dir\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.573445 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.573400 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-dshm\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.573445 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.573434 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-home\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.573593 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.573537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc58a75-4395-44a0-bd99-51a126a3f929-tls-certs\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.674744 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.674710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-home\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.674893 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.674763 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc58a75-4395-44a0-bd99-51a126a3f929-tls-certs\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.674893 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.674792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-model-cache\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.674893 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.674832 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh9kl\" (UniqueName: \"kubernetes.io/projected/edc58a75-4395-44a0-bd99-51a126a3f929-kube-api-access-nh9kl\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.674893 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.674862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-tmp-dir\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.675102 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.674999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-dshm\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.675201 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.675178 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-home\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.675283 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.675198 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-model-cache\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.675283 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.675242 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-tmp-dir\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.677114 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.677093 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-dshm\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.677283 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.677267 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc58a75-4395-44a0-bd99-51a126a3f929-tls-certs\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.683145 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.683126 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh9kl\" (UniqueName: \"kubernetes.io/projected/edc58a75-4395-44a0-bd99-51a126a3f929-kube-api-access-nh9kl\") pod \"test-model-982fc1-kserve-88c74c7f5-nwwzb\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.776716 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.776641 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:40.913228 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:40.913200 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb"] Jul 22 09:53:40.915331 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:53:40.915303 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedc58a75_4395_44a0_bd99_51a126a3f929.slice/crio-e6c0d16f244202c4d6bb752bfd4e388fd3e7591ea9e57cc5ad14d0a18a60391f WatchSource:0}: Error finding container e6c0d16f244202c4d6bb752bfd4e388fd3e7591ea9e57cc5ad14d0a18a60391f: Status 404 returned error can't find the container with id e6c0d16f244202c4d6bb752bfd4e388fd3e7591ea9e57cc5ad14d0a18a60391f Jul 22 09:53:41.307042 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:41.307008 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" event={"ID":"edc58a75-4395-44a0-bd99-51a126a3f929","Type":"ContainerStarted","Data":"3f2ce02daa45d371127fbb9b6f95f0dc14036fe2521288a0124273b3a0a9ba1d"} Jul 22 09:53:41.307042 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:41.307049 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" event={"ID":"edc58a75-4395-44a0-bd99-51a126a3f929","Type":"ContainerStarted","Data":"e6c0d16f244202c4d6bb752bfd4e388fd3e7591ea9e57cc5ad14d0a18a60391f"} Jul 22 09:53:41.344519 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:41.344464 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" podStartSLOduration=1.344448561 podStartE2EDuration="1.344448561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:53:41.342764425 +0000 UTC m=+2145.840271874" watchObservedRunningTime="2026-07-22 09:53:41.344448561 +0000 UTC m=+2145.841955996" Jul 22 09:53:45.251275 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:45.251238 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:53:50.777236 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:50.777145 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:50.777236 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:50.777200 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:50.789502 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:50.789466 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:51.364502 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:51.364476 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:53:56.187743 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:56.187715 2564 scope.go:117] "RemoveContainer" containerID="47e28eae8748a8fed1ccc45509fefddde7be6e724573083384d7acb5a4ff43f6" Jul 22 09:53:57.150088 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.150053 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww"] Jul 22 09:53:57.154751 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.154733 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.157189 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.157161 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b1d31b\"/\"openshift-service-ca.crt\"" Jul 22 09:53:57.157333 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.157315 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b1d31b\"/\"kube-root-ca.crt\"" Jul 22 09:53:57.158180 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.158157 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b1d31b\"/\"default-dockercfg-bjtcm\"" Jul 22 09:53:57.158277 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.158239 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b1d31b\"/\"test-model-b1d31b-kserve-self-signed-certs\"" Jul 22 09:53:57.163982 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.163962 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww"] Jul 22 09:53:57.225866 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.225823 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-home\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.226447 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.226423 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-model-cache\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.226542 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.226465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b6ed40-eaba-4719-a69d-b9602540287f-tls-certs\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.226542 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.226486 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-tmp-dir\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.226633 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.226610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtcvg\" (UniqueName: \"kubernetes.io/projected/a6b6ed40-eaba-4719-a69d-b9602540287f-kube-api-access-wtcvg\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.226734 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.226720 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-dshm\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.327753 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.327719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-dshm\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.327753 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.327756 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-home\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.328007 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.327790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-model-cache\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.328007 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.327820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b6ed40-eaba-4719-a69d-b9602540287f-tls-certs\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.328007 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.327845 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-tmp-dir\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.328007 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.327906 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtcvg\" (UniqueName: \"kubernetes.io/projected/a6b6ed40-eaba-4719-a69d-b9602540287f-kube-api-access-wtcvg\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.328222 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.328196 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-model-cache\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.328283 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.328261 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-home\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.328321 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.328307 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-tmp-dir\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.330063 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.330039 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-dshm\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.330244 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.330229 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b6ed40-eaba-4719-a69d-b9602540287f-tls-certs\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.337087 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.337065 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtcvg\" (UniqueName: \"kubernetes.io/projected/a6b6ed40-eaba-4719-a69d-b9602540287f-kube-api-access-wtcvg\") pod \"test-model-b1d31b-kserve-5c6999ddf9-ldbww\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.465115 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.465089 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:53:57.599170 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:57.599143 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww"] Jul 22 09:53:57.600658 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:53:57.600625 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6b6ed40_eaba_4719_a69d_b9602540287f.slice/crio-96046b369b17576c88b8aee660f32be25eeaa4d7b48094a9af9dd3d033b10e0c WatchSource:0}: Error finding container 96046b369b17576c88b8aee660f32be25eeaa4d7b48094a9af9dd3d033b10e0c: Status 404 returned error can't find the container with id 96046b369b17576c88b8aee660f32be25eeaa4d7b48094a9af9dd3d033b10e0c Jul 22 09:53:58.391012 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:58.390968 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" event={"ID":"a6b6ed40-eaba-4719-a69d-b9602540287f","Type":"ContainerStarted","Data":"56c6391732a8817f52d7152a17c3b98fa41ebe28f91a0af3152c97c9e6af9f2c"} Jul 22 09:53:58.391012 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:58.391008 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" event={"ID":"a6b6ed40-eaba-4719-a69d-b9602540287f","Type":"ContainerStarted","Data":"96046b369b17576c88b8aee660f32be25eeaa4d7b48094a9af9dd3d033b10e0c"} Jul 22 09:53:58.412507 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:53:58.412463 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" podStartSLOduration=1.412443453 podStartE2EDuration="1.412443453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:53:58.412354761 +0000 UTC m=+2162.909862198" watchObservedRunningTime="2026-07-22 09:53:58.412443453 +0000 UTC m=+2162.909950889" Jul 22 09:54:05.553531 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:54:05.553493 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:54:07.465772 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:54:07.465736 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:54:07.466232 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:54:07.465797 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:54:07.478383 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:54:07.478364 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:54:08.448835 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:54:08.448805 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:55:01.086640 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:01.086597 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb"] Jul 22 09:55:01.948776 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:01.948736 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww"] Jul 22 09:55:02.116069 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:02.116030 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8"] Jul 22 09:55:02.119333 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:02.119300 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8" Jul 22 09:55:02.122274 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:02.122245 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 09:55:02.122274 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:02.122265 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 09:55:02.130379 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:02.130347 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8"] Jul 22 09:55:02.251762 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:02.251647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c053ad0-0a99-4069-9c07-4ba9a55d8b01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8\" (UID: \"6c053ad0-0a99-4069-9c07-4ba9a55d8b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8" Jul 22 09:55:02.352861 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:02.352817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c053ad0-0a99-4069-9c07-4ba9a55d8b01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8\" (UID: \"6c053ad0-0a99-4069-9c07-4ba9a55d8b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8" Jul 22 09:55:02.353523 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:02.353495 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c053ad0-0a99-4069-9c07-4ba9a55d8b01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8\" (UID: \"6c053ad0-0a99-4069-9c07-4ba9a55d8b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8" Jul 22 09:55:02.432412 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:02.432371 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8" Jul 22 09:55:02.568744 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:02.568718 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8"] Jul 22 09:55:02.570767 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:55:02.570734 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c053ad0_0a99_4069_9c07_4ba9a55d8b01.slice/crio-9b9d64a1c15cc90e118341f46ca90d4f16078086d8a96f8509ab7f6bd0f0b216 WatchSource:0}: Error finding container 9b9d64a1c15cc90e118341f46ca90d4f16078086d8a96f8509ab7f6bd0f0b216: Status 404 returned error can't find the container with id 9b9d64a1c15cc90e118341f46ca90d4f16078086d8a96f8509ab7f6bd0f0b216 Jul 22 09:55:02.705535 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:02.705484 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8" event={"ID":"6c053ad0-0a99-4069-9c07-4ba9a55d8b01","Type":"ContainerStarted","Data":"9b9d64a1c15cc90e118341f46ca90d4f16078086d8a96f8509ab7f6bd0f0b216"} Jul 22 09:55:03.712207 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:03.712171 2564 generic.go:356] "Generic (PLEG): container finished" podID="6c053ad0-0a99-4069-9c07-4ba9a55d8b01" containerID="14f463d5cb67249ac30cbe139ad58cd35790b9967897a32eac9880aded0a12eb" exitCode=0 Jul 22 09:55:03.712598 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:03.712258 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8" event={"ID":"6c053ad0-0a99-4069-9c07-4ba9a55d8b01","Type":"ContainerDied","Data":"14f463d5cb67249ac30cbe139ad58cd35790b9967897a32eac9880aded0a12eb"} Jul 22 09:55:04.156143 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:04.156051 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:55:04.842403 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:04.842379 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8" Jul 22 09:55:04.979901 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:04.979801 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c053ad0-0a99-4069-9c07-4ba9a55d8b01-maas-api-service-ca\") pod \"6c053ad0-0a99-4069-9c07-4ba9a55d8b01\" (UID: \"6c053ad0-0a99-4069-9c07-4ba9a55d8b01\") " Jul 22 09:55:04.980138 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:04.980117 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c053ad0-0a99-4069-9c07-4ba9a55d8b01-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c053ad0-0a99-4069-9c07-4ba9a55d8b01" (UID: "6c053ad0-0a99-4069-9c07-4ba9a55d8b01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:55:04.980291 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:04.980270 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c053ad0-0a99-4069-9c07-4ba9a55d8b01-maas-api-service-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:55:05.723485 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:05.723454 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8" Jul 22 09:55:05.723660 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:05.723493 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8" event={"ID":"6c053ad0-0a99-4069-9c07-4ba9a55d8b01","Type":"ContainerDied","Data":"9b9d64a1c15cc90e118341f46ca90d4f16078086d8a96f8509ab7f6bd0f0b216"} Jul 22 09:55:05.723660 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:05.723541 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b9d64a1c15cc90e118341f46ca90d4f16078086d8a96f8509ab7f6bd0f0b216" Jul 22 09:55:13.853597 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:13.853558 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:55:16.106706 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.106658 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" podUID="edc58a75-4395-44a0-bd99-51a126a3f929" containerName="main" containerID="cri-o://3f2ce02daa45d371127fbb9b6f95f0dc14036fe2521288a0124273b3a0a9ba1d" gracePeriod=45 Jul 22 09:55:16.353926 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.353891 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:55:16.495310 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.495281 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-home\") pod \"edc58a75-4395-44a0-bd99-51a126a3f929\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " Jul 22 09:55:16.495487 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.495359 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-tmp-dir\") pod \"edc58a75-4395-44a0-bd99-51a126a3f929\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " Jul 22 09:55:16.495487 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.495381 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh9kl\" (UniqueName: \"kubernetes.io/projected/edc58a75-4395-44a0-bd99-51a126a3f929-kube-api-access-nh9kl\") pod \"edc58a75-4395-44a0-bd99-51a126a3f929\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " Jul 22 09:55:16.495487 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.495413 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc58a75-4395-44a0-bd99-51a126a3f929-tls-certs\") pod \"edc58a75-4395-44a0-bd99-51a126a3f929\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " Jul 22 09:55:16.495487 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.495447 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-model-cache\") pod \"edc58a75-4395-44a0-bd99-51a126a3f929\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " Jul 22 09:55:16.495689 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.495574 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-dshm\") pod \"edc58a75-4395-44a0-bd99-51a126a3f929\" (UID: \"edc58a75-4395-44a0-bd99-51a126a3f929\") " Jul 22 09:55:16.495689 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.495597 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-home" (OuterVolumeSpecName: "home") pod "edc58a75-4395-44a0-bd99-51a126a3f929" (UID: "edc58a75-4395-44a0-bd99-51a126a3f929"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:16.495689 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.495626 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "edc58a75-4395-44a0-bd99-51a126a3f929" (UID: "edc58a75-4395-44a0-bd99-51a126a3f929"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:16.495826 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.495711 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-model-cache" (OuterVolumeSpecName: "model-cache") pod "edc58a75-4395-44a0-bd99-51a126a3f929" (UID: "edc58a75-4395-44a0-bd99-51a126a3f929"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:16.495970 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.495926 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-home\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:55:16.496041 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.495977 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-tmp-dir\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:55:16.496041 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.496003 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-model-cache\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:55:16.497770 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.497744 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-dshm" (OuterVolumeSpecName: "dshm") pod "edc58a75-4395-44a0-bd99-51a126a3f929" (UID: "edc58a75-4395-44a0-bd99-51a126a3f929"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:16.497881 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.497797 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc58a75-4395-44a0-bd99-51a126a3f929-kube-api-access-nh9kl" (OuterVolumeSpecName: "kube-api-access-nh9kl") pod "edc58a75-4395-44a0-bd99-51a126a3f929" (UID: "edc58a75-4395-44a0-bd99-51a126a3f929"). InnerVolumeSpecName "kube-api-access-nh9kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:55:16.497881 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.497798 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc58a75-4395-44a0-bd99-51a126a3f929-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "edc58a75-4395-44a0-bd99-51a126a3f929" (UID: "edc58a75-4395-44a0-bd99-51a126a3f929"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:55:16.597039 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.596998 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh9kl\" (UniqueName: \"kubernetes.io/projected/edc58a75-4395-44a0-bd99-51a126a3f929-kube-api-access-nh9kl\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:55:16.597039 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.597028 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edc58a75-4395-44a0-bd99-51a126a3f929-tls-certs\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:55:16.597039 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.597044 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edc58a75-4395-44a0-bd99-51a126a3f929-dshm\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:55:16.777007 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.776901 2564 generic.go:356] "Generic (PLEG): container finished" podID="edc58a75-4395-44a0-bd99-51a126a3f929" containerID="3f2ce02daa45d371127fbb9b6f95f0dc14036fe2521288a0124273b3a0a9ba1d" exitCode=0 Jul 22 09:55:16.777007 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.776960 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" Jul 22 09:55:16.777234 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.776962 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" event={"ID":"edc58a75-4395-44a0-bd99-51a126a3f929","Type":"ContainerDied","Data":"3f2ce02daa45d371127fbb9b6f95f0dc14036fe2521288a0124273b3a0a9ba1d"} Jul 22 09:55:16.777234 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.777070 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb" event={"ID":"edc58a75-4395-44a0-bd99-51a126a3f929","Type":"ContainerDied","Data":"e6c0d16f244202c4d6bb752bfd4e388fd3e7591ea9e57cc5ad14d0a18a60391f"} Jul 22 09:55:16.777234 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.777087 2564 scope.go:117] "RemoveContainer" containerID="3f2ce02daa45d371127fbb9b6f95f0dc14036fe2521288a0124273b3a0a9ba1d" Jul 22 09:55:16.786396 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.786379 2564 scope.go:117] "RemoveContainer" containerID="3f2ce02daa45d371127fbb9b6f95f0dc14036fe2521288a0124273b3a0a9ba1d" Jul 22 09:55:16.786641 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:55:16.786619 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2ce02daa45d371127fbb9b6f95f0dc14036fe2521288a0124273b3a0a9ba1d\": container with ID starting with 3f2ce02daa45d371127fbb9b6f95f0dc14036fe2521288a0124273b3a0a9ba1d not found: ID does not exist" containerID="3f2ce02daa45d371127fbb9b6f95f0dc14036fe2521288a0124273b3a0a9ba1d" Jul 22 09:55:16.786734 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.786647 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2ce02daa45d371127fbb9b6f95f0dc14036fe2521288a0124273b3a0a9ba1d"} err="failed to get container status \"3f2ce02daa45d371127fbb9b6f95f0dc14036fe2521288a0124273b3a0a9ba1d\": rpc error: code = NotFound desc = could not find container \"3f2ce02daa45d371127fbb9b6f95f0dc14036fe2521288a0124273b3a0a9ba1d\": container with ID starting with 3f2ce02daa45d371127fbb9b6f95f0dc14036fe2521288a0124273b3a0a9ba1d not found: ID does not exist" Jul 22 09:55:16.798431 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.798407 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb"] Jul 22 09:55:16.802942 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.802914 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-982fc1/test-model-982fc1-kserve-88c74c7f5-nwwzb"] Jul 22 09:55:16.969160 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:16.969119 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" podUID="a6b6ed40-eaba-4719-a69d-b9602540287f" containerName="main" containerID="cri-o://56c6391732a8817f52d7152a17c3b98fa41ebe28f91a0af3152c97c9e6af9f2c" gracePeriod=45 Jul 22 09:55:17.214060 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.214012 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8"] Jul 22 09:55:17.216054 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.216025 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-982fc1-9vtt8"] Jul 22 09:55:17.218751 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.218736 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:55:17.404350 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.404241 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-tmp-dir\") pod \"a6b6ed40-eaba-4719-a69d-b9602540287f\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " Jul 22 09:55:17.404350 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.404299 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-model-cache\") pod \"a6b6ed40-eaba-4719-a69d-b9602540287f\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " Jul 22 09:55:17.404350 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.404350 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b6ed40-eaba-4719-a69d-b9602540287f-tls-certs\") pod \"a6b6ed40-eaba-4719-a69d-b9602540287f\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " Jul 22 09:55:17.404630 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.404371 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-dshm\") pod \"a6b6ed40-eaba-4719-a69d-b9602540287f\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " Jul 22 09:55:17.404630 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.404424 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-home\") pod \"a6b6ed40-eaba-4719-a69d-b9602540287f\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " Jul 22 09:55:17.404630 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.404446 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtcvg\" (UniqueName: \"kubernetes.io/projected/a6b6ed40-eaba-4719-a69d-b9602540287f-kube-api-access-wtcvg\") pod \"a6b6ed40-eaba-4719-a69d-b9602540287f\" (UID: \"a6b6ed40-eaba-4719-a69d-b9602540287f\") " Jul 22 09:55:17.404630 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.404578 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6b6ed40-eaba-4719-a69d-b9602540287f" (UID: "a6b6ed40-eaba-4719-a69d-b9602540287f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:17.404828 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.404659 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6b6ed40-eaba-4719-a69d-b9602540287f" (UID: "a6b6ed40-eaba-4719-a69d-b9602540287f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:17.404828 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.404668 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-home" (OuterVolumeSpecName: "home") pod "a6b6ed40-eaba-4719-a69d-b9602540287f" (UID: "a6b6ed40-eaba-4719-a69d-b9602540287f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:17.404828 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.404771 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-home\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:55:17.404828 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.404786 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-tmp-dir\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:55:17.404828 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.404795 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-model-cache\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:55:17.406756 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.406726 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-dshm" (OuterVolumeSpecName: "dshm") pod "a6b6ed40-eaba-4719-a69d-b9602540287f" (UID: "a6b6ed40-eaba-4719-a69d-b9602540287f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 09:55:17.406884 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.406785 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b6ed40-eaba-4719-a69d-b9602540287f-kube-api-access-wtcvg" (OuterVolumeSpecName: "kube-api-access-wtcvg") pod "a6b6ed40-eaba-4719-a69d-b9602540287f" (UID: "a6b6ed40-eaba-4719-a69d-b9602540287f"). InnerVolumeSpecName "kube-api-access-wtcvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 09:55:17.406884 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.406799 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b6ed40-eaba-4719-a69d-b9602540287f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6b6ed40-eaba-4719-a69d-b9602540287f" (UID: "a6b6ed40-eaba-4719-a69d-b9602540287f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 09:55:17.505775 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.505742 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtcvg\" (UniqueName: \"kubernetes.io/projected/a6b6ed40-eaba-4719-a69d-b9602540287f-kube-api-access-wtcvg\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:55:17.505775 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.505768 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b6ed40-eaba-4719-a69d-b9602540287f-tls-certs\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:55:17.505775 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.505780 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6b6ed40-eaba-4719-a69d-b9602540287f-dshm\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:55:17.783116 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.783085 2564 generic.go:356] "Generic (PLEG): container finished" podID="a6b6ed40-eaba-4719-a69d-b9602540287f" containerID="56c6391732a8817f52d7152a17c3b98fa41ebe28f91a0af3152c97c9e6af9f2c" exitCode=0 Jul 22 09:55:17.783296 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.783123 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" Jul 22 09:55:17.783296 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.783169 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" event={"ID":"a6b6ed40-eaba-4719-a69d-b9602540287f","Type":"ContainerDied","Data":"56c6391732a8817f52d7152a17c3b98fa41ebe28f91a0af3152c97c9e6af9f2c"} Jul 22 09:55:17.783296 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.783211 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww" event={"ID":"a6b6ed40-eaba-4719-a69d-b9602540287f","Type":"ContainerDied","Data":"96046b369b17576c88b8aee660f32be25eeaa4d7b48094a9af9dd3d033b10e0c"} Jul 22 09:55:17.783296 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.783233 2564 scope.go:117] "RemoveContainer" containerID="56c6391732a8817f52d7152a17c3b98fa41ebe28f91a0af3152c97c9e6af9f2c" Jul 22 09:55:17.792617 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.792599 2564 scope.go:117] "RemoveContainer" containerID="56c6391732a8817f52d7152a17c3b98fa41ebe28f91a0af3152c97c9e6af9f2c" Jul 22 09:55:17.792855 ip-10-0-134-55 kubenswrapper[2564]: E0722 09:55:17.792836 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c6391732a8817f52d7152a17c3b98fa41ebe28f91a0af3152c97c9e6af9f2c\": container with ID starting with 56c6391732a8817f52d7152a17c3b98fa41ebe28f91a0af3152c97c9e6af9f2c not found: ID does not exist" containerID="56c6391732a8817f52d7152a17c3b98fa41ebe28f91a0af3152c97c9e6af9f2c" Jul 22 09:55:17.792911 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.792863 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c6391732a8817f52d7152a17c3b98fa41ebe28f91a0af3152c97c9e6af9f2c"} err="failed to get container status \"56c6391732a8817f52d7152a17c3b98fa41ebe28f91a0af3152c97c9e6af9f2c\": rpc error: code = NotFound desc = could not find container \"56c6391732a8817f52d7152a17c3b98fa41ebe28f91a0af3152c97c9e6af9f2c\": container with ID starting with 56c6391732a8817f52d7152a17c3b98fa41ebe28f91a0af3152c97c9e6af9f2c not found: ID does not exist" Jul 22 09:55:17.806330 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.806304 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww"] Jul 22 09:55:17.809834 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:17.809814 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b1d31b/test-model-b1d31b-kserve-5c6999ddf9-ldbww"] Jul 22 09:55:18.014109 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:18.014075 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c053ad0-0a99-4069-9c07-4ba9a55d8b01" path="/var/lib/kubelet/pods/6c053ad0-0a99-4069-9c07-4ba9a55d8b01/volumes" Jul 22 09:55:18.014481 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:18.014468 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6b6ed40-eaba-4719-a69d-b9602540287f" path="/var/lib/kubelet/pods/a6b6ed40-eaba-4719-a69d-b9602540287f/volumes" Jul 22 09:55:18.014823 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:18.014811 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edc58a75-4395-44a0-bd99-51a126a3f929" path="/var/lib/kubelet/pods/edc58a75-4395-44a0-bd99-51a126a3f929/volumes" Jul 22 09:55:22.347810 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:22.347773 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:55:24.122834 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.122804 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5"] Jul 22 09:55:24.123378 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.123359 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6b6ed40-eaba-4719-a69d-b9602540287f" containerName="main" Jul 22 09:55:24.123520 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.123381 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6b6ed40-eaba-4719-a69d-b9602540287f" containerName="main" Jul 22 09:55:24.123520 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.123395 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c053ad0-0a99-4069-9c07-4ba9a55d8b01" containerName="revoke-keys" Jul 22 09:55:24.123520 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.123404 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c053ad0-0a99-4069-9c07-4ba9a55d8b01" containerName="revoke-keys" Jul 22 09:55:24.123520 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.123432 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc58a75-4395-44a0-bd99-51a126a3f929" containerName="main" Jul 22 09:55:24.123520 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.123440 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc58a75-4395-44a0-bd99-51a126a3f929" containerName="main" Jul 22 09:55:24.123760 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.123523 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc58a75-4395-44a0-bd99-51a126a3f929" containerName="main" Jul 22 09:55:24.123760 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.123535 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6b6ed40-eaba-4719-a69d-b9602540287f" containerName="main" Jul 22 09:55:24.123760 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.123546 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c053ad0-0a99-4069-9c07-4ba9a55d8b01" containerName="revoke-keys" Jul 22 09:55:24.128012 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.127992 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5" Jul 22 09:55:24.130259 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.130232 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 09:55:24.130358 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.130344 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 09:55:24.134385 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.134363 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5"] Jul 22 09:55:24.163736 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.163701 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7955bb-8695-4d9f-b4a5-238791237a86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5\" (UID: \"dd7955bb-8695-4d9f-b4a5-238791237a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5" Jul 22 09:55:24.264645 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.264602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7955bb-8695-4d9f-b4a5-238791237a86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5\" (UID: \"dd7955bb-8695-4d9f-b4a5-238791237a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5" Jul 22 09:55:24.265238 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.265220 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7955bb-8695-4d9f-b4a5-238791237a86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5\" (UID: \"dd7955bb-8695-4d9f-b4a5-238791237a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5" Jul 22 09:55:24.440085 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.440058 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5" Jul 22 09:55:24.586860 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.586831 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5"] Jul 22 09:55:24.588948 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:55:24.588909 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7955bb_8695_4d9f_b4a5_238791237a86.slice/crio-62d583dcd7bcaefb4f778346bcdb55c9f4195f5d76d8095bc39631d74b8f7352 WatchSource:0}: Error finding container 62d583dcd7bcaefb4f778346bcdb55c9f4195f5d76d8095bc39631d74b8f7352: Status 404 returned error can't find the container with id 62d583dcd7bcaefb4f778346bcdb55c9f4195f5d76d8095bc39631d74b8f7352 Jul 22 09:55:24.820975 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:24.820922 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5" event={"ID":"dd7955bb-8695-4d9f-b4a5-238791237a86","Type":"ContainerStarted","Data":"62d583dcd7bcaefb4f778346bcdb55c9f4195f5d76d8095bc39631d74b8f7352"} Jul 22 09:55:25.827331 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:25.827290 2564 generic.go:356] "Generic (PLEG): container finished" podID="dd7955bb-8695-4d9f-b4a5-238791237a86" containerID="703a59f6b6d32ee6bdc4b32d651ae889dc1448f40c887870a02a229569e3e5e4" exitCode=0 Jul 22 09:55:25.827764 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:25.827363 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5" event={"ID":"dd7955bb-8695-4d9f-b4a5-238791237a86","Type":"ContainerDied","Data":"703a59f6b6d32ee6bdc4b32d651ae889dc1448f40c887870a02a229569e3e5e4"} Jul 22 09:55:26.962272 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:26.962246 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5" Jul 22 09:55:26.989733 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:26.989711 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7955bb-8695-4d9f-b4a5-238791237a86-maas-api-service-ca\") pod \"dd7955bb-8695-4d9f-b4a5-238791237a86\" (UID: \"dd7955bb-8695-4d9f-b4a5-238791237a86\") " Jul 22 09:55:26.990058 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:26.990033 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7955bb-8695-4d9f-b4a5-238791237a86-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd7955bb-8695-4d9f-b4a5-238791237a86" (UID: "dd7955bb-8695-4d9f-b4a5-238791237a86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:55:27.091123 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:27.091041 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7955bb-8695-4d9f-b4a5-238791237a86-maas-api-service-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:55:27.838134 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:27.838105 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5" Jul 22 09:55:27.838288 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:27.838099 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5" event={"ID":"dd7955bb-8695-4d9f-b4a5-238791237a86","Type":"ContainerDied","Data":"62d583dcd7bcaefb4f778346bcdb55c9f4195f5d76d8095bc39631d74b8f7352"} Jul 22 09:55:27.838288 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:27.838229 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62d583dcd7bcaefb4f778346bcdb55c9f4195f5d76d8095bc39631d74b8f7352" Jul 22 09:55:37.451797 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:37.451764 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:55:39.279712 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:39.279676 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5"] Jul 22 09:55:39.283473 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:39.283446 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b1d31b-5z2n5"] Jul 22 09:55:40.014215 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:40.014177 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd7955bb-8695-4d9f-b4a5-238791237a86" path="/var/lib/kubelet/pods/dd7955bb-8695-4d9f-b4a5-238791237a86/volumes" Jul 22 09:55:56.213550 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:56.213518 2564 scope.go:117] "RemoveContainer" containerID="a8e9fd054dc7214fa7fbc6752f2234d5b76ee0446e177fbe43b9de007d93f43e" Jul 22 09:55:57.381310 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:55:57.381271 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:56:10.005746 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:10.005714 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r"] Jul 22 09:56:10.006235 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:10.006106 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd7955bb-8695-4d9f-b4a5-238791237a86" containerName="revoke-keys" Jul 22 09:56:10.006235 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:10.006118 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd7955bb-8695-4d9f-b4a5-238791237a86" containerName="revoke-keys" Jul 22 09:56:10.006235 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:10.006211 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd7955bb-8695-4d9f-b4a5-238791237a86" containerName="revoke-keys" Jul 22 09:56:10.009241 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:10.009221 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r" Jul 22 09:56:10.011821 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:10.011801 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 09:56:10.012161 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:10.012142 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 09:56:10.018354 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:10.018331 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r"] Jul 22 09:56:10.094352 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:10.094314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-86258d-4n27r\" (UID: \"eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r" Jul 22 09:56:10.195621 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:10.195588 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-86258d-4n27r\" (UID: \"eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r" Jul 22 09:56:10.196235 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:10.196217 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-86258d-4n27r\" (UID: \"eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r" Jul 22 09:56:10.320359 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:10.320261 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r" Jul 22 09:56:10.449660 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:10.449631 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r"] Jul 22 09:56:10.451427 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:56:10.451395 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea5c3e2_4ab1_40fa_a31b_cda6a03e5d28.slice/crio-4abd9f9b853d7e25e31b7ad3e2f8ca07810a9af107f6682918694ab5a2543eed WatchSource:0}: Error finding container 4abd9f9b853d7e25e31b7ad3e2f8ca07810a9af107f6682918694ab5a2543eed: Status 404 returned error can't find the container with id 4abd9f9b853d7e25e31b7ad3e2f8ca07810a9af107f6682918694ab5a2543eed Jul 22 09:56:11.056074 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:11.056041 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r" event={"ID":"eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28","Type":"ContainerStarted","Data":"a523ca863e55104d9f7ea6a8124d259a779c7e64c2fd95c977f9495bbc414db5"} Jul 22 09:56:11.056440 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:11.056091 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r" event={"ID":"eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28","Type":"ContainerStarted","Data":"4abd9f9b853d7e25e31b7ad3e2f8ca07810a9af107f6682918694ab5a2543eed"} Jul 22 09:56:11.073884 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:11.073831 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r" podStartSLOduration=2.073812294 podStartE2EDuration="2.073812294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:56:11.07314577 +0000 UTC m=+2295.570653206" watchObservedRunningTime="2026-07-22 09:56:11.073812294 +0000 UTC m=+2295.571319730" Jul 22 09:56:11.647263 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:11.647230 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:56:12.068729 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:12.068694 2564 generic.go:356] "Generic (PLEG): container finished" podID="eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28" containerID="a523ca863e55104d9f7ea6a8124d259a779c7e64c2fd95c977f9495bbc414db5" exitCode=0 Jul 22 09:56:12.069133 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:12.068770 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r" event={"ID":"eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28","Type":"ContainerDied","Data":"a523ca863e55104d9f7ea6a8124d259a779c7e64c2fd95c977f9495bbc414db5"} Jul 22 09:56:13.203919 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:13.203897 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r" Jul 22 09:56:13.323856 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:13.323813 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28-maas-api-service-ca\") pod \"eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28\" (UID: \"eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28\") " Jul 22 09:56:13.324152 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:13.324131 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28" (UID: "eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:56:13.425502 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:13.425426 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28-maas-api-service-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:56:14.079697 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:14.079665 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r" event={"ID":"eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28","Type":"ContainerDied","Data":"4abd9f9b853d7e25e31b7ad3e2f8ca07810a9af107f6682918694ab5a2543eed"} Jul 22 09:56:14.079847 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:14.079715 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4abd9f9b853d7e25e31b7ad3e2f8ca07810a9af107f6682918694ab5a2543eed" Jul 22 09:56:14.079847 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:14.079672 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r" Jul 22 09:56:28.956080 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:28.956045 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:56:30.093042 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:30.093010 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r"] Jul 22 09:56:30.098133 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:30.098106 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-86258d-4n27r"] Jul 22 09:56:32.015130 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:32.015100 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28" path="/var/lib/kubelet/pods/eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28/volumes" Jul 22 09:56:36.949744 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:36.949708 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr"] Jul 22 09:56:36.950133 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:36.950104 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28" containerName="revoke-keys" Jul 22 09:56:36.950133 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:36.950116 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28" containerName="revoke-keys" Jul 22 09:56:36.950265 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:36.950206 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eea5c3e2-4ab1-40fa-a31b-cda6a03e5d28" containerName="revoke-keys" Jul 22 09:56:36.953292 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:36.953276 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr" Jul 22 09:56:36.956199 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:36.956180 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 09:56:36.956315 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:36.956180 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 09:56:36.962735 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:36.962713 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr"] Jul 22 09:56:37.047704 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:37.047670 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78192c86-ff4f-4931-9e49-7183a2eb8fd3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr\" (UID: \"78192c86-ff4f-4931-9e49-7183a2eb8fd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr" Jul 22 09:56:37.149180 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:37.149150 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78192c86-ff4f-4931-9e49-7183a2eb8fd3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr\" (UID: \"78192c86-ff4f-4931-9e49-7183a2eb8fd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr" Jul 22 09:56:37.149836 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:37.149812 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78192c86-ff4f-4931-9e49-7183a2eb8fd3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr\" (UID: \"78192c86-ff4f-4931-9e49-7183a2eb8fd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr" Jul 22 09:56:37.264378 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:37.264303 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr" Jul 22 09:56:37.395253 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:37.395226 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr"] Jul 22 09:56:37.396658 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:56:37.396635 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78192c86_ff4f_4931_9e49_7183a2eb8fd3.slice/crio-6569b39a33828dbdd93ab3f32b5061643f0160eb88847a370ca1240cccc32059 WatchSource:0}: Error finding container 6569b39a33828dbdd93ab3f32b5061643f0160eb88847a370ca1240cccc32059: Status 404 returned error can't find the container with id 6569b39a33828dbdd93ab3f32b5061643f0160eb88847a370ca1240cccc32059 Jul 22 09:56:38.198460 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:38.198426 2564 generic.go:356] "Generic (PLEG): container finished" podID="78192c86-ff4f-4931-9e49-7183a2eb8fd3" containerID="716de8ec9cb4964e31eab16227aa2b7ed6b61df173129ac47afa870327fbdb65" exitCode=0 Jul 22 09:56:38.198813 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:38.198513 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr" event={"ID":"78192c86-ff4f-4931-9e49-7183a2eb8fd3","Type":"ContainerDied","Data":"716de8ec9cb4964e31eab16227aa2b7ed6b61df173129ac47afa870327fbdb65"} Jul 22 09:56:38.198813 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:38.198567 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr" event={"ID":"78192c86-ff4f-4931-9e49-7183a2eb8fd3","Type":"ContainerStarted","Data":"6569b39a33828dbdd93ab3f32b5061643f0160eb88847a370ca1240cccc32059"} Jul 22 09:56:39.329023 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:39.328997 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr" Jul 22 09:56:39.471253 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:39.471165 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78192c86-ff4f-4931-9e49-7183a2eb8fd3-maas-api-service-ca\") pod \"78192c86-ff4f-4931-9e49-7183a2eb8fd3\" (UID: \"78192c86-ff4f-4931-9e49-7183a2eb8fd3\") " Jul 22 09:56:39.471498 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:39.471477 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78192c86-ff4f-4931-9e49-7183a2eb8fd3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78192c86-ff4f-4931-9e49-7183a2eb8fd3" (UID: "78192c86-ff4f-4931-9e49-7183a2eb8fd3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 09:56:39.572104 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:39.572073 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78192c86-ff4f-4931-9e49-7183a2eb8fd3-maas-api-service-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 09:56:40.209017 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:40.208985 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr" Jul 22 09:56:40.209017 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:40.208997 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr" event={"ID":"78192c86-ff4f-4931-9e49-7183a2eb8fd3","Type":"ContainerDied","Data":"6569b39a33828dbdd93ab3f32b5061643f0160eb88847a370ca1240cccc32059"} Jul 22 09:56:40.209228 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:40.209039 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6569b39a33828dbdd93ab3f32b5061643f0160eb88847a370ca1240cccc32059" Jul 22 09:56:57.032265 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:57.032230 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr"] Jul 22 09:56:57.034113 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:57.034090 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5acdb-5l5nr"] Jul 22 09:56:57.058980 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:57.058954 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:56:58.014803 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:56:58.014770 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78192c86-ff4f-4931-9e49-7183a2eb8fd3" path="/var/lib/kubelet/pods/78192c86-ff4f-4931-9e49-7183a2eb8fd3/volumes" Jul 22 09:57:14.956446 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:14.956410 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:57:26.156130 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:26.156098 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:57:29.647860 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.647823 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj"] Jul 22 09:57:29.648459 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.648286 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78192c86-ff4f-4931-9e49-7183a2eb8fd3" containerName="revoke-keys" Jul 22 09:57:29.648459 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.648305 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="78192c86-ff4f-4931-9e49-7183a2eb8fd3" containerName="revoke-keys" Jul 22 09:57:29.648459 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.648424 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="78192c86-ff4f-4931-9e49-7183a2eb8fd3" containerName="revoke-keys" Jul 22 09:57:29.651630 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.651615 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.654188 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.654170 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-99612b\"/\"openshift-service-ca.crt\"" Jul 22 09:57:29.655223 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.655204 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99612b\"/\"default-dockercfg-qckx7\"" Jul 22 09:57:29.655320 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.655257 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-99612b\"/\"kube-root-ca.crt\"" Jul 22 09:57:29.655320 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.655300 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99612b\"/\"auth-test-model-99612b-kserve-self-signed-certs\"" Jul 22 09:57:29.660623 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.660592 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj"] Jul 22 09:57:29.738817 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.738786 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-tmp-dir\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.738997 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.738823 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-dshm\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.738997 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.738848 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-model-cache\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.738997 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.738973 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de253887-b48f-454f-a51f-0fa54e27d4d7-tls-certs\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.739131 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.739025 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-home\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.739131 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.739049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gk75\" (UniqueName: \"kubernetes.io/projected/de253887-b48f-454f-a51f-0fa54e27d4d7-kube-api-access-8gk75\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.839872 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.839842 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-tmp-dir\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.839872 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.839875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-dshm\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.840142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.839895 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-model-cache\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.840142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.840053 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de253887-b48f-454f-a51f-0fa54e27d4d7-tls-certs\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.840142 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.840127 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-home\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.840303 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.840159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gk75\" (UniqueName: \"kubernetes.io/projected/de253887-b48f-454f-a51f-0fa54e27d4d7-kube-api-access-8gk75\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.840361 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.840295 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-model-cache\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.840403 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.840355 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-tmp-dir\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.840494 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.840466 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-home\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.842261 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.842233 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-dshm\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.842423 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.842407 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de253887-b48f-454f-a51f-0fa54e27d4d7-tls-certs\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.847717 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.847695 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gk75\" (UniqueName: \"kubernetes.io/projected/de253887-b48f-454f-a51f-0fa54e27d4d7-kube-api-access-8gk75\") pod \"auth-test-model-99612b-kserve-67dd947499-xhkpj\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:29.961948 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:29.961914 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:30.098798 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:30.098769 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj"] Jul 22 09:57:30.099779 ip-10-0-134-55 kubenswrapper[2564]: W0722 09:57:30.099749 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde253887_b48f_454f_a51f_0fa54e27d4d7.slice/crio-5f3d68297885346473a89d088233cd484b7e2a09206e1713a50895f5a389c8e1 WatchSource:0}: Error finding container 5f3d68297885346473a89d088233cd484b7e2a09206e1713a50895f5a389c8e1: Status 404 returned error can't find the container with id 5f3d68297885346473a89d088233cd484b7e2a09206e1713a50895f5a389c8e1 Jul 22 09:57:30.460259 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:30.460229 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" event={"ID":"de253887-b48f-454f-a51f-0fa54e27d4d7","Type":"ContainerStarted","Data":"969afcff87434e5b05747d7c6c2730e084649a3ac775d5914102d6b0e8c003fc"} Jul 22 09:57:30.460461 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:30.460268 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" event={"ID":"de253887-b48f-454f-a51f-0fa54e27d4d7","Type":"ContainerStarted","Data":"5f3d68297885346473a89d088233cd484b7e2a09206e1713a50895f5a389c8e1"} Jul 22 09:57:30.479054 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:30.479000 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" podStartSLOduration=1.47898312 podStartE2EDuration="1.47898312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:57:30.478560727 +0000 UTC m=+2374.976068164" watchObservedRunningTime="2026-07-22 09:57:30.47898312 +0000 UTC m=+2374.976490535" Jul 22 09:57:34.755260 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:34.755223 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:57:39.962571 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:39.962528 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:39.962985 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:39.962590 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:39.975140 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:39.975119 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:40.520004 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:40.519974 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 09:57:48.039731 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.039701 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2"] Jul 22 09:57:48.043277 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.043261 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.045784 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.045763 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-88ceff\"/\"openshift-service-ca.crt\"" Jul 22 09:57:48.045899 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.045765 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-88ceff\"/\"default-dockercfg-4r4kr\"" Jul 22 09:57:48.046849 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.046830 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-88ceff\"/\"kube-root-ca.crt\"" Jul 22 09:57:48.046956 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.046875 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-88ceff\"/\"auth-test-model-88ceff-kserve-self-signed-certs\"" Jul 22 09:57:48.054002 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.053979 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2"] Jul 22 09:57:48.119625 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.119596 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-home\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.119750 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.119642 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-tmp-dir\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.119750 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.119673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj7j4\" (UniqueName: \"kubernetes.io/projected/545d530a-4a1f-430a-9b05-f7260177c28c-kube-api-access-cj7j4\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.119750 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.119718 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545d530a-4a1f-430a-9b05-f7260177c28c-tls-certs\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.119750 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.119737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-dshm\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.120134 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.119776 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-model-cache\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.220244 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.220209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-home\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.220414 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.220264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-tmp-dir\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.220414 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.220303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj7j4\" (UniqueName: \"kubernetes.io/projected/545d530a-4a1f-430a-9b05-f7260177c28c-kube-api-access-cj7j4\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.220414 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.220339 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545d530a-4a1f-430a-9b05-f7260177c28c-tls-certs\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.220595 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.220468 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-dshm\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.220595 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.220524 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-model-cache\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.220595 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.220603 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-home\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.220765 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.220662 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-tmp-dir\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.220817 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.220800 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-model-cache\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.222632 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.222612 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-dshm\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.222808 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.222790 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545d530a-4a1f-430a-9b05-f7260177c28c-tls-certs\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.229233 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.229212 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj7j4\" (UniqueName: \"kubernetes.io/projected/545d530a-4a1f-430a-9b05-f7260177c28c-kube-api-access-cj7j4\") pod \"auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.354738 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.354639 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:48.485449 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.485425 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2"] Jul 22 09:57:48.549922 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:48.549894 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" event={"ID":"545d530a-4a1f-430a-9b05-f7260177c28c","Type":"ContainerStarted","Data":"8d39f68e606498112ffbefdaca207a19bca85e02150d0f5a8d3645872585b280"} Jul 22 09:57:49.556099 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:49.556069 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" event={"ID":"545d530a-4a1f-430a-9b05-f7260177c28c","Type":"ContainerStarted","Data":"841dab2e84901111a3fdc19cd0a76109b1cc81a51c4d2b6c46245e05d7c1dc33"} Jul 22 09:57:49.577528 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:49.577465 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" podStartSLOduration=1.57744829 podStartE2EDuration="1.57744829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 09:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 09:57:49.576503452 +0000 UTC m=+2394.074010889" watchObservedRunningTime="2026-07-22 09:57:49.57744829 +0000 UTC m=+2394.074955728" Jul 22 09:57:52.057668 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:52.057633 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:57:56.094082 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:56.094052 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:57:56.100419 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:56.100399 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 09:57:58.355375 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:58.355337 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:58.355375 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:58.355386 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:58.367702 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:58.367675 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:57:58.610537 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:57:58.610460 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 09:58:08.659818 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:58:08.659785 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:58:28.062253 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:58:28.062175 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:58:48.064762 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:58:48.064727 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:59:04.771836 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:59:04.771799 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:59:24.160735 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:59:24.160700 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:59:43.665747 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:59:43.665716 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 09:59:59.284032 ip-10-0-134-55 kubenswrapper[2564]: I0722 09:59:59.283991 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj"] Jul 22 10:00:00.138735 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.138696 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745240-wsbb8"] Jul 22 10:00:00.142500 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.142476 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-88ceff-29745240-5mwvs"] Jul 22 10:00:00.142641 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.142596 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745240-wsbb8" Jul 22 10:00:00.145339 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.145319 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 10:00:00.145747 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.145730 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99612b-29745240-d7tg8"] Jul 22 10:00:00.145850 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.145834 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-88ceff-29745240-5mwvs" Jul 22 10:00:00.149083 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.149063 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99612b-29745240-d7tg8" Jul 22 10:00:00.156321 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.156297 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745240-wsbb8" Jul 22 10:00:00.156549 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.156529 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745240-wsbb8"] Jul 22 10:00:00.158127 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.158102 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-88ceff-29745240-5mwvs"] Jul 22 10:00:00.163445 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.163406 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99612b-29745240-d7tg8"] Jul 22 10:00:00.167367 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.167341 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-88ceff-29745240-5mwvs" Jul 22 10:00:00.201778 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.201756 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99612b-29745240-d7tg8" Jul 22 10:00:00.340305 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.338493 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2"] Jul 22 10:00:00.344250 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.344224 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745240-wsbb8"] Jul 22 10:00:00.345159 ip-10-0-134-55 kubenswrapper[2564]: W0722 10:00:00.345123 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04284088_dea8_46e3_bd1b_8633c768f845.slice/crio-c5cae5581aa89d6e08b2b8d2b8a98c827b0a6f73a53a7a1b8745a31f207137d5 WatchSource:0}: Error finding container c5cae5581aa89d6e08b2b8d2b8a98c827b0a6f73a53a7a1b8745a31f207137d5: Status 404 returned error can't find the container with id c5cae5581aa89d6e08b2b8d2b8a98c827b0a6f73a53a7a1b8745a31f207137d5 Jul 22 10:00:00.566429 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.566400 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-88ceff-29745240-5mwvs"] Jul 22 10:00:00.568751 ip-10-0-134-55 kubenswrapper[2564]: W0722 10:00:00.568725 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26368474_2cec_4b99_b369_7497b52015b4.slice/crio-f605b93e315796cc71424858a0b1e010e316279a6333d07b3c51cb2fbc2a6729 WatchSource:0}: Error finding container f605b93e315796cc71424858a0b1e010e316279a6333d07b3c51cb2fbc2a6729: Status 404 returned error can't find the container with id f605b93e315796cc71424858a0b1e010e316279a6333d07b3c51cb2fbc2a6729 Jul 22 10:00:00.596907 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:00.596885 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99612b-29745240-d7tg8"] Jul 22 10:00:00.604008 ip-10-0-134-55 kubenswrapper[2564]: W0722 10:00:00.603985 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ee02aa_2d47_4e48_a95a_e279a150444f.slice/crio-b73af3b36510feb207a8902f005d9761052a4bbdcf103942016c7efe2c3ed386 WatchSource:0}: Error finding container b73af3b36510feb207a8902f005d9761052a4bbdcf103942016c7efe2c3ed386: Status 404 returned error can't find the container with id b73af3b36510feb207a8902f005d9761052a4bbdcf103942016c7efe2c3ed386 Jul 22 10:00:01.215200 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:01.215161 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745240-wsbb8" event={"ID":"04284088-dea8-46e3-bd1b-8633c768f845","Type":"ContainerStarted","Data":"7d19c4bf5fe47fde74f3b23cda6c687280362fe9c07d7090befe83e2e20d88c0"} Jul 22 10:00:01.215387 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:01.215218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745240-wsbb8" event={"ID":"04284088-dea8-46e3-bd1b-8633c768f845","Type":"ContainerStarted","Data":"c5cae5581aa89d6e08b2b8d2b8a98c827b0a6f73a53a7a1b8745a31f207137d5"} Jul 22 10:00:01.216619 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:01.216598 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99612b-29745240-d7tg8" event={"ID":"18ee02aa-2d47-4e48-a95a-e279a150444f","Type":"ContainerStarted","Data":"6ac0afe39de4f64ba52fab00636841c33d187d135e67d812657af6c56577acac"} Jul 22 10:00:01.216679 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:01.216636 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99612b-29745240-d7tg8" event={"ID":"18ee02aa-2d47-4e48-a95a-e279a150444f","Type":"ContainerStarted","Data":"b73af3b36510feb207a8902f005d9761052a4bbdcf103942016c7efe2c3ed386"} Jul 22 10:00:01.217984 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:01.217960 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-88ceff-29745240-5mwvs" event={"ID":"26368474-2cec-4b99-b369-7497b52015b4","Type":"ContainerStarted","Data":"ef250027e92b24501fe3a701c9165fda69a328530c97a5454d88958efd13ce21"} Jul 22 10:00:01.218073 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:01.218001 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-88ceff-29745240-5mwvs" event={"ID":"26368474-2cec-4b99-b369-7497b52015b4","Type":"ContainerStarted","Data":"f605b93e315796cc71424858a0b1e010e316279a6333d07b3c51cb2fbc2a6729"} Jul 22 10:00:01.241738 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:01.241695 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745240-wsbb8" podStartSLOduration=1.241680964 podStartE2EDuration="1.241680964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 10:00:01.240522683 +0000 UTC m=+2525.738030119" watchObservedRunningTime="2026-07-22 10:00:01.241680964 +0000 UTC m=+2525.739188399" Jul 22 10:00:01.290820 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:01.290766 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-88ceff-29745240-5mwvs" podStartSLOduration=1.290750451 podStartE2EDuration="1.290750451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 10:00:01.290356231 +0000 UTC m=+2525.787863668" watchObservedRunningTime="2026-07-22 10:00:01.290750451 +0000 UTC m=+2525.788257887" Jul 22 10:00:01.291837 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:01.291799 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99612b-29745240-d7tg8" podStartSLOduration=1.2917900279999999 podStartE2EDuration="1.291790028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 10:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 10:00:01.264594309 +0000 UTC m=+2525.762101744" watchObservedRunningTime="2026-07-22 10:00:01.291790028 +0000 UTC m=+2525.789297511" Jul 22 10:00:02.224211 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:02.224171 2564 generic.go:356] "Generic (PLEG): container finished" podID="26368474-2cec-4b99-b369-7497b52015b4" containerID="ef250027e92b24501fe3a701c9165fda69a328530c97a5454d88958efd13ce21" exitCode=0 Jul 22 10:00:02.224693 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:02.224255 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-88ceff-29745240-5mwvs" event={"ID":"26368474-2cec-4b99-b369-7497b52015b4","Type":"ContainerDied","Data":"ef250027e92b24501fe3a701c9165fda69a328530c97a5454d88958efd13ce21"} Jul 22 10:00:02.225702 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:02.225676 2564 generic.go:356] "Generic (PLEG): container finished" podID="04284088-dea8-46e3-bd1b-8633c768f845" containerID="7d19c4bf5fe47fde74f3b23cda6c687280362fe9c07d7090befe83e2e20d88c0" exitCode=0 Jul 22 10:00:02.225822 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:02.225758 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745240-wsbb8" event={"ID":"04284088-dea8-46e3-bd1b-8633c768f845","Type":"ContainerDied","Data":"7d19c4bf5fe47fde74f3b23cda6c687280362fe9c07d7090befe83e2e20d88c0"} Jul 22 10:00:02.227035 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:02.227015 2564 generic.go:356] "Generic (PLEG): container finished" podID="18ee02aa-2d47-4e48-a95a-e279a150444f" containerID="6ac0afe39de4f64ba52fab00636841c33d187d135e67d812657af6c56577acac" exitCode=0 Jul 22 10:00:02.227125 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:02.227058 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99612b-29745240-d7tg8" event={"ID":"18ee02aa-2d47-4e48-a95a-e279a150444f","Type":"ContainerDied","Data":"6ac0afe39de4f64ba52fab00636841c33d187d135e67d812657af6c56577acac"} Jul 22 10:00:02.961300 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:02.961258 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 10:00:03.417811 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:03.417782 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99612b-29745240-d7tg8" Jul 22 10:00:03.420699 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:03.420678 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-88ceff-29745240-5mwvs" Jul 22 10:00:03.424291 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:03.424273 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745240-wsbb8" Jul 22 10:00:04.239678 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:04.239644 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745240-wsbb8" Jul 22 10:00:04.239678 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:04.239653 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745240-wsbb8" event={"ID":"04284088-dea8-46e3-bd1b-8633c768f845","Type":"ContainerDied","Data":"c5cae5581aa89d6e08b2b8d2b8a98c827b0a6f73a53a7a1b8745a31f207137d5"} Jul 22 10:00:04.239909 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:04.239704 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5cae5581aa89d6e08b2b8d2b8a98c827b0a6f73a53a7a1b8745a31f207137d5" Jul 22 10:00:04.240993 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:04.240973 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99612b-29745240-d7tg8" Jul 22 10:00:04.241118 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:04.241008 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99612b-29745240-d7tg8" event={"ID":"18ee02aa-2d47-4e48-a95a-e279a150444f","Type":"ContainerDied","Data":"b73af3b36510feb207a8902f005d9761052a4bbdcf103942016c7efe2c3ed386"} Jul 22 10:00:04.241118 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:04.241046 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b73af3b36510feb207a8902f005d9761052a4bbdcf103942016c7efe2c3ed386" Jul 22 10:00:04.242416 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:04.242391 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-88ceff-29745240-5mwvs" event={"ID":"26368474-2cec-4b99-b369-7497b52015b4","Type":"ContainerDied","Data":"f605b93e315796cc71424858a0b1e010e316279a6333d07b3c51cb2fbc2a6729"} Jul 22 10:00:04.242416 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:04.242412 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-88ceff-29745240-5mwvs" Jul 22 10:00:04.242561 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:04.242427 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f605b93e315796cc71424858a0b1e010e316279a6333d07b3c51cb2fbc2a6729" Jul 22 10:00:14.303777 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.303712 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" podUID="de253887-b48f-454f-a51f-0fa54e27d4d7" containerName="main" containerID="cri-o://969afcff87434e5b05747d7c6c2730e084649a3ac775d5914102d6b0e8c003fc" gracePeriod=45 Jul 22 10:00:14.544856 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.544832 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 10:00:14.575607 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.575534 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gk75\" (UniqueName: \"kubernetes.io/projected/de253887-b48f-454f-a51f-0fa54e27d4d7-kube-api-access-8gk75\") pod \"de253887-b48f-454f-a51f-0fa54e27d4d7\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " Jul 22 10:00:14.575739 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.575626 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-dshm\") pod \"de253887-b48f-454f-a51f-0fa54e27d4d7\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " Jul 22 10:00:14.575739 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.575650 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de253887-b48f-454f-a51f-0fa54e27d4d7-tls-certs\") pod \"de253887-b48f-454f-a51f-0fa54e27d4d7\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " Jul 22 10:00:14.575739 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.575674 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-tmp-dir\") pod \"de253887-b48f-454f-a51f-0fa54e27d4d7\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " Jul 22 10:00:14.575739 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.575724 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-home\") pod \"de253887-b48f-454f-a51f-0fa54e27d4d7\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " Jul 22 10:00:14.575992 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.575784 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-model-cache\") pod \"de253887-b48f-454f-a51f-0fa54e27d4d7\" (UID: \"de253887-b48f-454f-a51f-0fa54e27d4d7\") " Jul 22 10:00:14.576064 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.576004 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-home" (OuterVolumeSpecName: "home") pod "de253887-b48f-454f-a51f-0fa54e27d4d7" (UID: "de253887-b48f-454f-a51f-0fa54e27d4d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:00:14.576064 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.576018 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de253887-b48f-454f-a51f-0fa54e27d4d7" (UID: "de253887-b48f-454f-a51f-0fa54e27d4d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:00:14.576176 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.576165 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-tmp-dir\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:00:14.576233 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.576182 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-home\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:00:14.576233 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.576176 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "de253887-b48f-454f-a51f-0fa54e27d4d7" (UID: "de253887-b48f-454f-a51f-0fa54e27d4d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:00:14.577875 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.577849 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-dshm" (OuterVolumeSpecName: "dshm") pod "de253887-b48f-454f-a51f-0fa54e27d4d7" (UID: "de253887-b48f-454f-a51f-0fa54e27d4d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:00:14.578012 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.577929 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de253887-b48f-454f-a51f-0fa54e27d4d7-kube-api-access-8gk75" (OuterVolumeSpecName: "kube-api-access-8gk75") pod "de253887-b48f-454f-a51f-0fa54e27d4d7" (UID: "de253887-b48f-454f-a51f-0fa54e27d4d7"). InnerVolumeSpecName "kube-api-access-8gk75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 10:00:14.578103 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.578020 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de253887-b48f-454f-a51f-0fa54e27d4d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de253887-b48f-454f-a51f-0fa54e27d4d7" (UID: "de253887-b48f-454f-a51f-0fa54e27d4d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 10:00:14.677044 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.677013 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-dshm\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:00:14.677044 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.677039 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de253887-b48f-454f-a51f-0fa54e27d4d7-tls-certs\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:00:14.677044 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.677049 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de253887-b48f-454f-a51f-0fa54e27d4d7-model-cache\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:00:14.677271 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:14.677057 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gk75\" (UniqueName: \"kubernetes.io/projected/de253887-b48f-454f-a51f-0fa54e27d4d7-kube-api-access-8gk75\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:00:15.307314 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.307271 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" event={"ID":"de253887-b48f-454f-a51f-0fa54e27d4d7","Type":"ContainerDied","Data":"969afcff87434e5b05747d7c6c2730e084649a3ac775d5914102d6b0e8c003fc"} Jul 22 10:00:15.307314 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.307310 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" Jul 22 10:00:15.307822 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.307274 2564 generic.go:356] "Generic (PLEG): container finished" podID="de253887-b48f-454f-a51f-0fa54e27d4d7" containerID="969afcff87434e5b05747d7c6c2730e084649a3ac775d5914102d6b0e8c003fc" exitCode=0 Jul 22 10:00:15.307822 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.307379 2564 scope.go:117] "RemoveContainer" containerID="969afcff87434e5b05747d7c6c2730e084649a3ac775d5914102d6b0e8c003fc" Jul 22 10:00:15.307822 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.307411 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj" event={"ID":"de253887-b48f-454f-a51f-0fa54e27d4d7","Type":"ContainerDied","Data":"5f3d68297885346473a89d088233cd484b7e2a09206e1713a50895f5a389c8e1"} Jul 22 10:00:15.317122 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.317103 2564 scope.go:117] "RemoveContainer" containerID="969afcff87434e5b05747d7c6c2730e084649a3ac775d5914102d6b0e8c003fc" Jul 22 10:00:15.317380 ip-10-0-134-55 kubenswrapper[2564]: E0722 10:00:15.317360 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"969afcff87434e5b05747d7c6c2730e084649a3ac775d5914102d6b0e8c003fc\": container with ID starting with 969afcff87434e5b05747d7c6c2730e084649a3ac775d5914102d6b0e8c003fc not found: ID does not exist" containerID="969afcff87434e5b05747d7c6c2730e084649a3ac775d5914102d6b0e8c003fc" Jul 22 10:00:15.317435 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.317386 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"969afcff87434e5b05747d7c6c2730e084649a3ac775d5914102d6b0e8c003fc"} err="failed to get container status \"969afcff87434e5b05747d7c6c2730e084649a3ac775d5914102d6b0e8c003fc\": rpc error: code = NotFound desc = could not find container \"969afcff87434e5b05747d7c6c2730e084649a3ac775d5914102d6b0e8c003fc\": container with ID starting with 969afcff87434e5b05747d7c6c2730e084649a3ac775d5914102d6b0e8c003fc not found: ID does not exist" Jul 22 10:00:15.330691 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.330667 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj"] Jul 22 10:00:15.335217 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.335197 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-99612b/auth-test-model-99612b-kserve-67dd947499-xhkpj"] Jul 22 10:00:15.364338 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.364313 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" podUID="545d530a-4a1f-430a-9b05-f7260177c28c" containerName="main" containerID="cri-o://841dab2e84901111a3fdc19cd0a76109b1cc81a51c4d2b6c46245e05d7c1dc33" gracePeriod=45 Jul 22 10:00:15.614354 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.614325 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 10:00:15.688182 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.688155 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-home\") pod \"545d530a-4a1f-430a-9b05-f7260177c28c\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " Jul 22 10:00:15.688348 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.688212 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-tmp-dir\") pod \"545d530a-4a1f-430a-9b05-f7260177c28c\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " Jul 22 10:00:15.688348 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.688244 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545d530a-4a1f-430a-9b05-f7260177c28c-tls-certs\") pod \"545d530a-4a1f-430a-9b05-f7260177c28c\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " Jul 22 10:00:15.688348 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.688279 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj7j4\" (UniqueName: \"kubernetes.io/projected/545d530a-4a1f-430a-9b05-f7260177c28c-kube-api-access-cj7j4\") pod \"545d530a-4a1f-430a-9b05-f7260177c28c\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " Jul 22 10:00:15.688348 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.688318 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-dshm\") pod \"545d530a-4a1f-430a-9b05-f7260177c28c\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " Jul 22 10:00:15.688584 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.688371 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-model-cache\") pod \"545d530a-4a1f-430a-9b05-f7260177c28c\" (UID: \"545d530a-4a1f-430a-9b05-f7260177c28c\") " Jul 22 10:00:15.688584 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.688436 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-home" (OuterVolumeSpecName: "home") pod "545d530a-4a1f-430a-9b05-f7260177c28c" (UID: "545d530a-4a1f-430a-9b05-f7260177c28c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:00:15.688584 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.688516 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "545d530a-4a1f-430a-9b05-f7260177c28c" (UID: "545d530a-4a1f-430a-9b05-f7260177c28c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:00:15.688716 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.688691 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-model-cache" (OuterVolumeSpecName: "model-cache") pod "545d530a-4a1f-430a-9b05-f7260177c28c" (UID: "545d530a-4a1f-430a-9b05-f7260177c28c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:00:15.688777 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.688761 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-home\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:00:15.688816 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.688781 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-tmp-dir\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:00:15.690534 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.690502 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-dshm" (OuterVolumeSpecName: "dshm") pod "545d530a-4a1f-430a-9b05-f7260177c28c" (UID: "545d530a-4a1f-430a-9b05-f7260177c28c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:00:15.690632 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.690554 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/545d530a-4a1f-430a-9b05-f7260177c28c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "545d530a-4a1f-430a-9b05-f7260177c28c" (UID: "545d530a-4a1f-430a-9b05-f7260177c28c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 10:00:15.690632 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.690575 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545d530a-4a1f-430a-9b05-f7260177c28c-kube-api-access-cj7j4" (OuterVolumeSpecName: "kube-api-access-cj7j4") pod "545d530a-4a1f-430a-9b05-f7260177c28c" (UID: "545d530a-4a1f-430a-9b05-f7260177c28c"). InnerVolumeSpecName "kube-api-access-cj7j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 10:00:15.790039 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.790003 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/545d530a-4a1f-430a-9b05-f7260177c28c-tls-certs\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:00:15.790039 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.790034 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj7j4\" (UniqueName: \"kubernetes.io/projected/545d530a-4a1f-430a-9b05-f7260177c28c-kube-api-access-cj7j4\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:00:15.790039 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.790044 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-dshm\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:00:15.790338 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:15.790055 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/545d530a-4a1f-430a-9b05-f7260177c28c-model-cache\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:00:16.015187 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:16.015155 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de253887-b48f-454f-a51f-0fa54e27d4d7" path="/var/lib/kubelet/pods/de253887-b48f-454f-a51f-0fa54e27d4d7/volumes" Jul 22 10:00:16.313574 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:16.313471 2564 generic.go:356] "Generic (PLEG): container finished" podID="545d530a-4a1f-430a-9b05-f7260177c28c" containerID="841dab2e84901111a3fdc19cd0a76109b1cc81a51c4d2b6c46245e05d7c1dc33" exitCode=0 Jul 22 10:00:16.313574 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:16.313518 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" Jul 22 10:00:16.314121 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:16.313565 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" event={"ID":"545d530a-4a1f-430a-9b05-f7260177c28c","Type":"ContainerDied","Data":"841dab2e84901111a3fdc19cd0a76109b1cc81a51c4d2b6c46245e05d7c1dc33"} Jul 22 10:00:16.314121 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:16.313617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2" event={"ID":"545d530a-4a1f-430a-9b05-f7260177c28c","Type":"ContainerDied","Data":"8d39f68e606498112ffbefdaca207a19bca85e02150d0f5a8d3645872585b280"} Jul 22 10:00:16.314121 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:16.313640 2564 scope.go:117] "RemoveContainer" containerID="841dab2e84901111a3fdc19cd0a76109b1cc81a51c4d2b6c46245e05d7c1dc33" Jul 22 10:00:16.322147 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:16.322130 2564 scope.go:117] "RemoveContainer" containerID="841dab2e84901111a3fdc19cd0a76109b1cc81a51c4d2b6c46245e05d7c1dc33" Jul 22 10:00:16.322378 ip-10-0-134-55 kubenswrapper[2564]: E0722 10:00:16.322360 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841dab2e84901111a3fdc19cd0a76109b1cc81a51c4d2b6c46245e05d7c1dc33\": container with ID starting with 841dab2e84901111a3fdc19cd0a76109b1cc81a51c4d2b6c46245e05d7c1dc33 not found: ID does not exist" containerID="841dab2e84901111a3fdc19cd0a76109b1cc81a51c4d2b6c46245e05d7c1dc33" Jul 22 10:00:16.322443 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:16.322386 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841dab2e84901111a3fdc19cd0a76109b1cc81a51c4d2b6c46245e05d7c1dc33"} err="failed to get container status \"841dab2e84901111a3fdc19cd0a76109b1cc81a51c4d2b6c46245e05d7c1dc33\": rpc error: code = NotFound desc = could not find container \"841dab2e84901111a3fdc19cd0a76109b1cc81a51c4d2b6c46245e05d7c1dc33\": container with ID starting with 841dab2e84901111a3fdc19cd0a76109b1cc81a51c4d2b6c46245e05d7c1dc33 not found: ID does not exist" Jul 22 10:00:16.333629 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:16.333607 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2"] Jul 22 10:00:16.338475 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:16.338450 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/auth-test-model-88ceff-kserve-5cbb8bddf5-klxf2"] Jul 22 10:00:18.014401 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:18.014368 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="545d530a-4a1f-430a-9b05-f7260177c28c" path="/var/lib/kubelet/pods/545d530a-4a1f-430a-9b05-f7260177c28c/volumes" Jul 22 10:00:32.983364 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983333 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv"] Jul 22 10:00:32.983800 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983727 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de253887-b48f-454f-a51f-0fa54e27d4d7" containerName="main" Jul 22 10:00:32.983800 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983740 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de253887-b48f-454f-a51f-0fa54e27d4d7" containerName="main" Jul 22 10:00:32.983800 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983750 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545d530a-4a1f-430a-9b05-f7260177c28c" containerName="main" Jul 22 10:00:32.983800 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983755 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="545d530a-4a1f-430a-9b05-f7260177c28c" containerName="main" Jul 22 10:00:32.983800 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983776 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18ee02aa-2d47-4e48-a95a-e279a150444f" containerName="cleanup" Jul 22 10:00:32.983800 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983781 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="18ee02aa-2d47-4e48-a95a-e279a150444f" containerName="cleanup" Jul 22 10:00:32.983800 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983790 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26368474-2cec-4b99-b369-7497b52015b4" containerName="cleanup" Jul 22 10:00:32.983800 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983797 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="26368474-2cec-4b99-b369-7497b52015b4" containerName="cleanup" Jul 22 10:00:32.984083 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983808 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04284088-dea8-46e3-bd1b-8633c768f845" containerName="cleanup" Jul 22 10:00:32.984083 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983816 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="04284088-dea8-46e3-bd1b-8633c768f845" containerName="cleanup" Jul 22 10:00:32.984083 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983869 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="545d530a-4a1f-430a-9b05-f7260177c28c" containerName="main" Jul 22 10:00:32.984083 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983877 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="18ee02aa-2d47-4e48-a95a-e279a150444f" containerName="cleanup" Jul 22 10:00:32.984083 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983887 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de253887-b48f-454f-a51f-0fa54e27d4d7" containerName="main" Jul 22 10:00:32.984083 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983896 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="26368474-2cec-4b99-b369-7497b52015b4" containerName="cleanup" Jul 22 10:00:32.984083 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.983902 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="04284088-dea8-46e3-bd1b-8633c768f845" containerName="cleanup" Jul 22 10:00:32.986812 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.986796 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:32.990337 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.990315 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-99612b\"/\"kube-root-ca.crt\"" Jul 22 10:00:32.990337 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.990331 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99612b\"/\"sub-test-model-99612b-kserve-self-signed-certs\"" Jul 22 10:00:32.990552 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.990352 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-99612b\"/\"openshift-service-ca.crt\"" Jul 22 10:00:32.990552 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.990331 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99612b\"/\"default-dockercfg-qckx7\"" Jul 22 10:00:32.996174 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:32.996154 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv"] Jul 22 10:00:33.061021 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.060992 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c28818-50d6-4f38-816e-e33a61e1b899-tls-certs\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.061177 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.061027 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-home\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.061177 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.061086 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-model-cache\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.061177 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.061119 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp7bm\" (UniqueName: \"kubernetes.io/projected/e2c28818-50d6-4f38-816e-e33a61e1b899-kube-api-access-hp7bm\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.061305 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.061179 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-dshm\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.061305 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.061215 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-tmp-dir\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.161710 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.161678 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-tmp-dir\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.161886 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.161750 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c28818-50d6-4f38-816e-e33a61e1b899-tls-certs\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.161886 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.161771 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-home\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.161886 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.161797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-model-cache\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.161886 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.161815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp7bm\" (UniqueName: \"kubernetes.io/projected/e2c28818-50d6-4f38-816e-e33a61e1b899-kube-api-access-hp7bm\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.162134 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.162029 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-dshm\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.162134 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.162108 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-tmp-dir\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.162237 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.162151 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-model-cache\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.162237 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.162223 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-home\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.164009 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.163987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-dshm\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.164113 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.164095 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c28818-50d6-4f38-816e-e33a61e1b899-tls-certs\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.172252 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.172231 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp7bm\" (UniqueName: \"kubernetes.io/projected/e2c28818-50d6-4f38-816e-e33a61e1b899-kube-api-access-hp7bm\") pod \"sub-test-model-99612b-kserve-6f585c7b65-g4dvv\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.296066 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.295983 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:33.437835 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:33.437771 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv"] Jul 22 10:00:33.440206 ip-10-0-134-55 kubenswrapper[2564]: W0722 10:00:33.440169 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2c28818_50d6_4f38_816e_e33a61e1b899.slice/crio-3101fc24fe19b8e416bd898b16671fe3cbb2228bddda9f39797ed4d1b268346d WatchSource:0}: Error finding container 3101fc24fe19b8e416bd898b16671fe3cbb2228bddda9f39797ed4d1b268346d: Status 404 returned error can't find the container with id 3101fc24fe19b8e416bd898b16671fe3cbb2228bddda9f39797ed4d1b268346d Jul 22 10:00:34.409400 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:34.409361 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" event={"ID":"e2c28818-50d6-4f38-816e-e33a61e1b899","Type":"ContainerStarted","Data":"936b19895203dfaf33f040a6ab06dac0ec51cd6def641b2a5c785de1196bf902"} Jul 22 10:00:34.409801 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:34.409409 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" event={"ID":"e2c28818-50d6-4f38-816e-e33a61e1b899","Type":"ContainerStarted","Data":"3101fc24fe19b8e416bd898b16671fe3cbb2228bddda9f39797ed4d1b268346d"} Jul 22 10:00:34.428742 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:34.428699 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" podStartSLOduration=2.428685126 podStartE2EDuration="2.428685126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 10:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 10:00:34.427421307 +0000 UTC m=+2558.924928744" watchObservedRunningTime="2026-07-22 10:00:34.428685126 +0000 UTC m=+2558.926192561" Jul 22 10:00:36.360349 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:36.360311 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 10:00:43.296870 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:43.296818 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:43.296870 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:43.296885 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:43.309473 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:43.309448 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:43.466607 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:43.466576 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:00:55.254136 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.254100 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb"] Jul 22 10:00:55.257864 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.257846 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.261401 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.261377 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-88ceff\"/\"openshift-service-ca.crt\"" Jul 22 10:00:55.261401 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.261388 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-88ceff\"/\"sub-test-model-88ceff-kserve-self-signed-certs\"" Jul 22 10:00:55.261587 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.261388 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-88ceff\"/\"default-dockercfg-4r4kr\"" Jul 22 10:00:55.261587 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.261379 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-88ceff\"/\"kube-root-ca.crt\"" Jul 22 10:00:55.270189 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.270165 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb"] Jul 22 10:00:55.365312 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.365279 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-home\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.365312 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.365312 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-tls-certs\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.365513 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.365334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdk6t\" (UniqueName: \"kubernetes.io/projected/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-kube-api-access-cdk6t\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.365513 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.365446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-dshm\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.365513 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.365475 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-tmp-dir\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.365513 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.365506 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-model-cache\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.466799 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.466767 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-model-cache\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.466896 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.466824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-home\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.466896 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.466850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-tls-certs\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.466896 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.466882 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdk6t\" (UniqueName: \"kubernetes.io/projected/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-kube-api-access-cdk6t\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.467060 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.466984 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-dshm\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.467060 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.467011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-tmp-dir\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.467261 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.467234 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-model-cache\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.467335 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.467273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-home\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.467335 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.467329 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-tmp-dir\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.469193 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.469171 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-dshm\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.469494 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.469473 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-tls-certs\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.476684 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.476656 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdk6t\" (UniqueName: \"kubernetes.io/projected/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-kube-api-access-cdk6t\") pod \"sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.567818 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.567737 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:00:55.699891 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:55.699863 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb"] Jul 22 10:00:55.701552 ip-10-0-134-55 kubenswrapper[2564]: W0722 10:00:55.701510 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39bb0377_6025_4a3e_8bd7_32dcc41a8e3c.slice/crio-23a75d8a0477985ba18ef7f6dc7a66f708c6d3fd71972163e0f271c699b5e2e9 WatchSource:0}: Error finding container 23a75d8a0477985ba18ef7f6dc7a66f708c6d3fd71972163e0f271c699b5e2e9: Status 404 returned error can't find the container with id 23a75d8a0477985ba18ef7f6dc7a66f708c6d3fd71972163e0f271c699b5e2e9 Jul 22 10:00:56.521995 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:56.521958 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" event={"ID":"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c","Type":"ContainerStarted","Data":"437f97eda0a123735e253998ae470db0edb95465b87c5b18063488ad3b4204d6"} Jul 22 10:00:56.521995 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:56.522000 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" event={"ID":"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c","Type":"ContainerStarted","Data":"23a75d8a0477985ba18ef7f6dc7a66f708c6d3fd71972163e0f271c699b5e2e9"} Jul 22 10:00:56.540997 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:56.540859 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" podStartSLOduration=1.540845285 podStartE2EDuration="1.540845285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 10:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 10:00:56.540550895 +0000 UTC m=+2581.038058325" watchObservedRunningTime="2026-07-22 10:00:56.540845285 +0000 UTC m=+2581.038352721" Jul 22 10:00:59.049821 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:00:59.049785 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 10:01:05.568069 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:05.568038 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:01:05.568537 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:05.568091 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:01:05.586522 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:05.586496 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:01:06.583228 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:06.583200 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:01:20.862430 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:20.862345 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 10:01:34.260024 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:34.259990 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 10:01:41.161948 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:41.161899 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv"] Jul 22 10:01:42.052973 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:42.052929 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb"] Jul 22 10:01:50.263000 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.262962 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp"] Jul 22 10:01:50.266740 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.266722 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.269229 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.269207 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99612b\"/\"rate-test-model-99612b-kserve-self-signed-certs\"" Jul 22 10:01:50.276478 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.276459 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp"] Jul 22 10:01:50.372027 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.371993 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s7xz\" (UniqueName: \"kubernetes.io/projected/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-kube-api-access-8s7xz\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.372194 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.372044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-tmp-dir\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.372194 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.372120 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-tls-certs\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.372194 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.372169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-dshm\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.372307 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.372197 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-model-cache\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.372307 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.372234 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-home\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.473411 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.473359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-dshm\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.473411 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.473422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-model-cache\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.473684 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.473470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-home\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.473684 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.473507 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s7xz\" (UniqueName: \"kubernetes.io/projected/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-kube-api-access-8s7xz\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.473684 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.473559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-tmp-dir\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.473684 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.473604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-tls-certs\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.473857 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.473816 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-model-cache\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.473924 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.473858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-home\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.473924 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.473897 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-tmp-dir\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.475776 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.475755 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-dshm\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.476014 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.475997 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-tls-certs\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.482386 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.482368 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s7xz\" (UniqueName: \"kubernetes.io/projected/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-kube-api-access-8s7xz\") pod \"rate-test-model-99612b-kserve-6884889567-q6kwp\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.577392 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.577303 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:01:50.707512 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.707489 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp"] Jul 22 10:01:50.709794 ip-10-0-134-55 kubenswrapper[2564]: W0722 10:01:50.709765 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod358f985d_b5a8_49f9_ae02_ef52b2fd21f8.slice/crio-91e5ba4a3c4ba7ce9463a61896844ae0b9eea50434c371f4dd39d4cf6129c39e WatchSource:0}: Error finding container 91e5ba4a3c4ba7ce9463a61896844ae0b9eea50434c371f4dd39d4cf6129c39e: Status 404 returned error can't find the container with id 91e5ba4a3c4ba7ce9463a61896844ae0b9eea50434c371f4dd39d4cf6129c39e Jul 22 10:01:50.808207 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.808172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" event={"ID":"358f985d-b5a8-49f9-ae02-ef52b2fd21f8","Type":"ContainerStarted","Data":"ee9d03ece0d116a98f58adfb8fd6cb20609c116c74f3555c6692ea7132425d1d"} Jul 22 10:01:50.808371 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.808217 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" event={"ID":"358f985d-b5a8-49f9-ae02-ef52b2fd21f8","Type":"ContainerStarted","Data":"91e5ba4a3c4ba7ce9463a61896844ae0b9eea50434c371f4dd39d4cf6129c39e"} Jul 22 10:01:50.829970 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:50.829864 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" podStartSLOduration=0.829849671 podStartE2EDuration="829.849671ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 10:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 10:01:50.828370528 +0000 UTC m=+2635.325877964" watchObservedRunningTime="2026-07-22 10:01:50.829849671 +0000 UTC m=+2635.327357107" Jul 22 10:01:53.154272 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:53.154243 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 10:01:56.181780 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.181723 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" podUID="e2c28818-50d6-4f38-816e-e33a61e1b899" containerName="main" containerID="cri-o://936b19895203dfaf33f040a6ab06dac0ec51cd6def641b2a5c785de1196bf902" gracePeriod=45 Jul 22 10:01:56.289674 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.289648 2564 scope.go:117] "RemoveContainer" containerID="703a59f6b6d32ee6bdc4b32d651ae889dc1448f40c887870a02a229569e3e5e4" Jul 22 10:01:56.297750 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.297730 2564 scope.go:117] "RemoveContainer" containerID="14f463d5cb67249ac30cbe139ad58cd35790b9967897a32eac9880aded0a12eb" Jul 22 10:01:56.437477 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.437455 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:01:56.530349 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.530321 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-dshm\") pod \"e2c28818-50d6-4f38-816e-e33a61e1b899\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " Jul 22 10:01:56.530522 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.530378 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-home\") pod \"e2c28818-50d6-4f38-816e-e33a61e1b899\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " Jul 22 10:01:56.530522 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.530398 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c28818-50d6-4f38-816e-e33a61e1b899-tls-certs\") pod \"e2c28818-50d6-4f38-816e-e33a61e1b899\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " Jul 22 10:01:56.530522 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.530415 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-model-cache\") pod \"e2c28818-50d6-4f38-816e-e33a61e1b899\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " Jul 22 10:01:56.530522 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.530494 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-tmp-dir\") pod \"e2c28818-50d6-4f38-816e-e33a61e1b899\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " Jul 22 10:01:56.530744 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.530536 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp7bm\" (UniqueName: \"kubernetes.io/projected/e2c28818-50d6-4f38-816e-e33a61e1b899-kube-api-access-hp7bm\") pod \"e2c28818-50d6-4f38-816e-e33a61e1b899\" (UID: \"e2c28818-50d6-4f38-816e-e33a61e1b899\") " Jul 22 10:01:56.530744 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.530680 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-home" (OuterVolumeSpecName: "home") pod "e2c28818-50d6-4f38-816e-e33a61e1b899" (UID: "e2c28818-50d6-4f38-816e-e33a61e1b899"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:01:56.530744 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.530708 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2c28818-50d6-4f38-816e-e33a61e1b899" (UID: "e2c28818-50d6-4f38-816e-e33a61e1b899"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:01:56.530850 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.530753 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2c28818-50d6-4f38-816e-e33a61e1b899" (UID: "e2c28818-50d6-4f38-816e-e33a61e1b899"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:01:56.530919 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.530894 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-home\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:01:56.531045 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.530921 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-model-cache\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:01:56.531045 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.530959 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-tmp-dir\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:01:56.532512 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.532488 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2c28818-50d6-4f38-816e-e33a61e1b899-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2c28818-50d6-4f38-816e-e33a61e1b899" (UID: "e2c28818-50d6-4f38-816e-e33a61e1b899"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 10:01:56.532594 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.532509 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-dshm" (OuterVolumeSpecName: "dshm") pod "e2c28818-50d6-4f38-816e-e33a61e1b899" (UID: "e2c28818-50d6-4f38-816e-e33a61e1b899"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:01:56.532594 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.532513 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c28818-50d6-4f38-816e-e33a61e1b899-kube-api-access-hp7bm" (OuterVolumeSpecName: "kube-api-access-hp7bm") pod "e2c28818-50d6-4f38-816e-e33a61e1b899" (UID: "e2c28818-50d6-4f38-816e-e33a61e1b899"). InnerVolumeSpecName "kube-api-access-hp7bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 10:01:56.631475 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.631441 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c28818-50d6-4f38-816e-e33a61e1b899-tls-certs\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:01:56.631475 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.631468 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp7bm\" (UniqueName: \"kubernetes.io/projected/e2c28818-50d6-4f38-816e-e33a61e1b899-kube-api-access-hp7bm\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:01:56.631475 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.631481 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2c28818-50d6-4f38-816e-e33a61e1b899-dshm\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:01:56.838548 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.838458 2564 generic.go:356] "Generic (PLEG): container finished" podID="e2c28818-50d6-4f38-816e-e33a61e1b899" containerID="936b19895203dfaf33f040a6ab06dac0ec51cd6def641b2a5c785de1196bf902" exitCode=0 Jul 22 10:01:56.838548 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.838496 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" Jul 22 10:01:56.838548 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.838522 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" event={"ID":"e2c28818-50d6-4f38-816e-e33a61e1b899","Type":"ContainerDied","Data":"936b19895203dfaf33f040a6ab06dac0ec51cd6def641b2a5c785de1196bf902"} Jul 22 10:01:56.838803 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.838551 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv" event={"ID":"e2c28818-50d6-4f38-816e-e33a61e1b899","Type":"ContainerDied","Data":"3101fc24fe19b8e416bd898b16671fe3cbb2228bddda9f39797ed4d1b268346d"} Jul 22 10:01:56.838803 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.838568 2564 scope.go:117] "RemoveContainer" containerID="936b19895203dfaf33f040a6ab06dac0ec51cd6def641b2a5c785de1196bf902" Jul 22 10:01:56.848104 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.848086 2564 scope.go:117] "RemoveContainer" containerID="936b19895203dfaf33f040a6ab06dac0ec51cd6def641b2a5c785de1196bf902" Jul 22 10:01:56.848362 ip-10-0-134-55 kubenswrapper[2564]: E0722 10:01:56.848343 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936b19895203dfaf33f040a6ab06dac0ec51cd6def641b2a5c785de1196bf902\": container with ID starting with 936b19895203dfaf33f040a6ab06dac0ec51cd6def641b2a5c785de1196bf902 not found: ID does not exist" containerID="936b19895203dfaf33f040a6ab06dac0ec51cd6def641b2a5c785de1196bf902" Jul 22 10:01:56.848432 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.848378 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936b19895203dfaf33f040a6ab06dac0ec51cd6def641b2a5c785de1196bf902"} err="failed to get container status \"936b19895203dfaf33f040a6ab06dac0ec51cd6def641b2a5c785de1196bf902\": rpc error: code = NotFound desc = could not find container \"936b19895203dfaf33f040a6ab06dac0ec51cd6def641b2a5c785de1196bf902\": container with ID starting with 936b19895203dfaf33f040a6ab06dac0ec51cd6def641b2a5c785de1196bf902 not found: ID does not exist" Jul 22 10:01:56.860394 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.860313 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv"] Jul 22 10:01:56.862473 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:56.862450 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-99612b/sub-test-model-99612b-kserve-6f585c7b65-g4dvv"] Jul 22 10:01:57.072343 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.072304 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" podUID="39bb0377-6025-4a3e-8bd7-32dcc41a8e3c" containerName="main" containerID="cri-o://437f97eda0a123735e253998ae470db0edb95465b87c5b18063488ad3b4204d6" gracePeriod=45 Jul 22 10:01:57.321779 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.321756 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:01:57.438866 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.438772 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-tmp-dir\") pod \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " Jul 22 10:01:57.438866 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.438854 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-home\") pod \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " Jul 22 10:01:57.439128 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.438883 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdk6t\" (UniqueName: \"kubernetes.io/projected/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-kube-api-access-cdk6t\") pod \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " Jul 22 10:01:57.439128 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.438908 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-model-cache\") pod \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " Jul 22 10:01:57.439128 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.439018 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-tls-certs\") pod \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " Jul 22 10:01:57.439128 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.439066 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-dshm\") pod \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\" (UID: \"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c\") " Jul 22 10:01:57.439339 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.439141 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39bb0377-6025-4a3e-8bd7-32dcc41a8e3c" (UID: "39bb0377-6025-4a3e-8bd7-32dcc41a8e3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:01:57.439339 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.439139 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-home" (OuterVolumeSpecName: "home") pod "39bb0377-6025-4a3e-8bd7-32dcc41a8e3c" (UID: "39bb0377-6025-4a3e-8bd7-32dcc41a8e3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:01:57.439339 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.439181 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-model-cache" (OuterVolumeSpecName: "model-cache") pod "39bb0377-6025-4a3e-8bd7-32dcc41a8e3c" (UID: "39bb0377-6025-4a3e-8bd7-32dcc41a8e3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:01:57.439483 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.439393 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-tmp-dir\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:01:57.439483 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.439408 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-home\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:01:57.439483 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.439424 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-model-cache\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:01:57.440964 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.440930 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-dshm" (OuterVolumeSpecName: "dshm") pod "39bb0377-6025-4a3e-8bd7-32dcc41a8e3c" (UID: "39bb0377-6025-4a3e-8bd7-32dcc41a8e3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:01:57.441073 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.441025 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39bb0377-6025-4a3e-8bd7-32dcc41a8e3c" (UID: "39bb0377-6025-4a3e-8bd7-32dcc41a8e3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 10:01:57.441119 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.441101 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-kube-api-access-cdk6t" (OuterVolumeSpecName: "kube-api-access-cdk6t") pod "39bb0377-6025-4a3e-8bd7-32dcc41a8e3c" (UID: "39bb0377-6025-4a3e-8bd7-32dcc41a8e3c"). InnerVolumeSpecName "kube-api-access-cdk6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 10:01:57.540097 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.540065 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdk6t\" (UniqueName: \"kubernetes.io/projected/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-kube-api-access-cdk6t\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:01:57.540097 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.540089 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-tls-certs\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:01:57.540097 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.540100 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c-dshm\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:01:57.846472 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.846384 2564 generic.go:356] "Generic (PLEG): container finished" podID="39bb0377-6025-4a3e-8bd7-32dcc41a8e3c" containerID="437f97eda0a123735e253998ae470db0edb95465b87c5b18063488ad3b4204d6" exitCode=0 Jul 22 10:01:57.846472 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.846424 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" Jul 22 10:01:57.846690 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.846470 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" event={"ID":"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c","Type":"ContainerDied","Data":"437f97eda0a123735e253998ae470db0edb95465b87c5b18063488ad3b4204d6"} Jul 22 10:01:57.846690 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.846513 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb" event={"ID":"39bb0377-6025-4a3e-8bd7-32dcc41a8e3c","Type":"ContainerDied","Data":"23a75d8a0477985ba18ef7f6dc7a66f708c6d3fd71972163e0f271c699b5e2e9"} Jul 22 10:01:57.846690 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.846531 2564 scope.go:117] "RemoveContainer" containerID="437f97eda0a123735e253998ae470db0edb95465b87c5b18063488ad3b4204d6" Jul 22 10:01:57.855513 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.855498 2564 scope.go:117] "RemoveContainer" containerID="437f97eda0a123735e253998ae470db0edb95465b87c5b18063488ad3b4204d6" Jul 22 10:01:57.855756 ip-10-0-134-55 kubenswrapper[2564]: E0722 10:01:57.855738 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437f97eda0a123735e253998ae470db0edb95465b87c5b18063488ad3b4204d6\": container with ID starting with 437f97eda0a123735e253998ae470db0edb95465b87c5b18063488ad3b4204d6 not found: ID does not exist" containerID="437f97eda0a123735e253998ae470db0edb95465b87c5b18063488ad3b4204d6" Jul 22 10:01:57.855797 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.855766 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437f97eda0a123735e253998ae470db0edb95465b87c5b18063488ad3b4204d6"} err="failed to get container status \"437f97eda0a123735e253998ae470db0edb95465b87c5b18063488ad3b4204d6\": rpc error: code = NotFound desc = could not find container \"437f97eda0a123735e253998ae470db0edb95465b87c5b18063488ad3b4204d6\": container with ID starting with 437f97eda0a123735e253998ae470db0edb95465b87c5b18063488ad3b4204d6 not found: ID does not exist" Jul 22 10:01:57.870968 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.869819 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb"] Jul 22 10:01:57.876811 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:57.876787 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/sub-test-model-88ceff-kserve-7b7ccf584c-lwhsb"] Jul 22 10:01:58.014781 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:58.014746 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39bb0377-6025-4a3e-8bd7-32dcc41a8e3c" path="/var/lib/kubelet/pods/39bb0377-6025-4a3e-8bd7-32dcc41a8e3c/volumes" Jul 22 10:01:58.015197 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:01:58.015181 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2c28818-50d6-4f38-816e-e33a61e1b899" path="/var/lib/kubelet/pods/e2c28818-50d6-4f38-816e-e33a61e1b899/volumes" Jul 22 10:02:00.577765 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:00.577727 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:02:00.577765 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:00.577774 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:02:00.590366 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:00.590342 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:02:00.873830 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:00.873750 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:02:15.770985 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.770954 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m"] Jul 22 10:02:15.771365 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.771352 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c28818-50d6-4f38-816e-e33a61e1b899" containerName="main" Jul 22 10:02:15.771409 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.771366 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c28818-50d6-4f38-816e-e33a61e1b899" containerName="main" Jul 22 10:02:15.771409 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.771379 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39bb0377-6025-4a3e-8bd7-32dcc41a8e3c" containerName="main" Jul 22 10:02:15.771409 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.771385 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="39bb0377-6025-4a3e-8bd7-32dcc41a8e3c" containerName="main" Jul 22 10:02:15.771506 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.771447 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2c28818-50d6-4f38-816e-e33a61e1b899" containerName="main" Jul 22 10:02:15.771506 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.771457 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="39bb0377-6025-4a3e-8bd7-32dcc41a8e3c" containerName="main" Jul 22 10:02:15.774482 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.774467 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.777248 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.777219 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-88ceff\"/\"kube-root-ca.crt\"" Jul 22 10:02:15.778457 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.778437 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-88ceff\"/\"openshift-service-ca.crt\"" Jul 22 10:02:15.778457 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.778447 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-88ceff\"/\"default-dockercfg-4r4kr\"" Jul 22 10:02:15.778684 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.778470 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-88ceff\"/\"rate-test-model-88ceff-kserve-self-signed-certs\"" Jul 22 10:02:15.788863 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.788836 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m"] Jul 22 10:02:15.804144 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.804045 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-tls-certs\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.804144 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.804106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-tmp-dir\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.804345 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.804146 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-dshm\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.804345 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.804191 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-model-cache\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.804345 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.804296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-home\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.804507 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.804365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cpxf\" (UniqueName: \"kubernetes.io/projected/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-kube-api-access-6cpxf\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.905323 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.905284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-tls-certs\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.905323 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.905329 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-tmp-dir\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.905549 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.905355 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-dshm\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.905549 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.905380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-model-cache\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.905549 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.905433 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-home\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.905549 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.905499 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cpxf\" (UniqueName: \"kubernetes.io/projected/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-kube-api-access-6cpxf\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.905760 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.905684 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-tmp-dir\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.905819 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.905806 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-model-cache\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.905885 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.905867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-home\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.907697 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.907677 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-dshm\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.907866 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.907847 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-tls-certs\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:15.915167 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:15.915145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cpxf\" (UniqueName: \"kubernetes.io/projected/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-kube-api-access-6cpxf\") pod \"rate-test-model-88ceff-kserve-6854d74f6c-7944m\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:16.083273 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:16.083189 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:16.219102 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:16.219072 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m"] Jul 22 10:02:16.220731 ip-10-0-134-55 kubenswrapper[2564]: W0722 10:02:16.220695 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c3bc65e_19f8_4205_bb54_fc8fc90d389e.slice/crio-37107478cc6e8b27b7c7a0cb26c486d51fb12343bc4d184f5e0c3c07b08a45b1 WatchSource:0}: Error finding container 37107478cc6e8b27b7c7a0cb26c486d51fb12343bc4d184f5e0c3c07b08a45b1: Status 404 returned error can't find the container with id 37107478cc6e8b27b7c7a0cb26c486d51fb12343bc4d184f5e0c3c07b08a45b1 Jul 22 10:02:16.944238 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:16.944204 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" event={"ID":"3c3bc65e-19f8-4205-bb54-fc8fc90d389e","Type":"ContainerStarted","Data":"6ad966655e8ab1ba3b4853821773a040df74915877f8b9326f782a3cf0b43d58"} Jul 22 10:02:16.944696 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:16.944249 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" event={"ID":"3c3bc65e-19f8-4205-bb54-fc8fc90d389e","Type":"ContainerStarted","Data":"37107478cc6e8b27b7c7a0cb26c486d51fb12343bc4d184f5e0c3c07b08a45b1"} Jul 22 10:02:16.964749 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:16.964704 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" podStartSLOduration=1.96468913 podStartE2EDuration="1.96468913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 10:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 10:02:16.963125583 +0000 UTC m=+2661.460633021" watchObservedRunningTime="2026-07-22 10:02:16.96468913 +0000 UTC m=+2661.462196566" Jul 22 10:02:19.060387 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:19.060351 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 10:02:26.083520 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:26.083491 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:26.083999 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:26.083720 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:26.095871 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:26.095846 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:27.003391 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:27.003363 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:02:36.273662 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:36.273631 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 10:02:51.258848 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:51.258766 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 10:02:56.126951 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:56.126911 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 10:02:56.134051 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:56.134033 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mdr8h_90be7f2f-8c43-40b8-9126-11e4f0f72f96/ovn-acl-logging/0.log" Jul 22 10:02:56.321235 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:56.321201 2564 scope.go:117] "RemoveContainer" containerID="716de8ec9cb4964e31eab16227aa2b7ed6b61df173129ac47afa870327fbdb65" Jul 22 10:02:56.329359 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:56.329340 2564 scope.go:117] "RemoveContainer" containerID="a523ca863e55104d9f7ea6a8124d259a779c7e64c2fd95c977f9495bbc414db5" Jul 22 10:02:58.556407 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:58.556380 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp"] Jul 22 10:02:59.402013 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:02:59.401983 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m"] Jul 22 10:03:06.562058 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:06.562021 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 10:03:13.576071 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.575997 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" podUID="358f985d-b5a8-49f9-ae02-ef52b2fd21f8" containerName="main" containerID="cri-o://ee9d03ece0d116a98f58adfb8fd6cb20609c116c74f3555c6692ea7132425d1d" gracePeriod=45 Jul 22 10:03:13.831384 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.831320 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:03:13.951719 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.951682 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-tls-certs\") pod \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " Jul 22 10:03:13.951885 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.951750 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-home\") pod \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " Jul 22 10:03:13.951885 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.951786 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-dshm\") pod \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " Jul 22 10:03:13.951885 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.951842 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-tmp-dir\") pod \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " Jul 22 10:03:13.951885 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.951879 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s7xz\" (UniqueName: \"kubernetes.io/projected/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-kube-api-access-8s7xz\") pod \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " Jul 22 10:03:13.952111 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.951908 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-model-cache\") pod \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\" (UID: \"358f985d-b5a8-49f9-ae02-ef52b2fd21f8\") " Jul 22 10:03:13.952111 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.952021 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-home" (OuterVolumeSpecName: "home") pod "358f985d-b5a8-49f9-ae02-ef52b2fd21f8" (UID: "358f985d-b5a8-49f9-ae02-ef52b2fd21f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:03:13.952226 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.952176 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "358f985d-b5a8-49f9-ae02-ef52b2fd21f8" (UID: "358f985d-b5a8-49f9-ae02-ef52b2fd21f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:03:13.952226 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.952212 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "358f985d-b5a8-49f9-ae02-ef52b2fd21f8" (UID: "358f985d-b5a8-49f9-ae02-ef52b2fd21f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:03:13.952339 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.952323 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-home\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:03:13.952377 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.952341 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-tmp-dir\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:03:13.952377 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.952355 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-model-cache\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:03:13.953911 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.953885 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-dshm" (OuterVolumeSpecName: "dshm") pod "358f985d-b5a8-49f9-ae02-ef52b2fd21f8" (UID: "358f985d-b5a8-49f9-ae02-ef52b2fd21f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:03:13.953911 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.953892 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "358f985d-b5a8-49f9-ae02-ef52b2fd21f8" (UID: "358f985d-b5a8-49f9-ae02-ef52b2fd21f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 10:03:13.954081 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:13.953970 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-kube-api-access-8s7xz" (OuterVolumeSpecName: "kube-api-access-8s7xz") pod "358f985d-b5a8-49f9-ae02-ef52b2fd21f8" (UID: "358f985d-b5a8-49f9-ae02-ef52b2fd21f8"). InnerVolumeSpecName "kube-api-access-8s7xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 10:03:14.053131 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.053102 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s7xz\" (UniqueName: \"kubernetes.io/projected/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-kube-api-access-8s7xz\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:03:14.053268 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.053129 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-tls-certs\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:03:14.053268 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.053148 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358f985d-b5a8-49f9-ae02-ef52b2fd21f8-dshm\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:03:14.226955 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.226902 2564 generic.go:356] "Generic (PLEG): container finished" podID="358f985d-b5a8-49f9-ae02-ef52b2fd21f8" containerID="ee9d03ece0d116a98f58adfb8fd6cb20609c116c74f3555c6692ea7132425d1d" exitCode=0 Jul 22 10:03:14.227142 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.226961 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" Jul 22 10:03:14.227142 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.226989 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" event={"ID":"358f985d-b5a8-49f9-ae02-ef52b2fd21f8","Type":"ContainerDied","Data":"ee9d03ece0d116a98f58adfb8fd6cb20609c116c74f3555c6692ea7132425d1d"} Jul 22 10:03:14.227142 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.227035 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp" event={"ID":"358f985d-b5a8-49f9-ae02-ef52b2fd21f8","Type":"ContainerDied","Data":"91e5ba4a3c4ba7ce9463a61896844ae0b9eea50434c371f4dd39d4cf6129c39e"} Jul 22 10:03:14.227142 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.227052 2564 scope.go:117] "RemoveContainer" containerID="ee9d03ece0d116a98f58adfb8fd6cb20609c116c74f3555c6692ea7132425d1d" Jul 22 10:03:14.235714 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.235699 2564 scope.go:117] "RemoveContainer" containerID="ee9d03ece0d116a98f58adfb8fd6cb20609c116c74f3555c6692ea7132425d1d" Jul 22 10:03:14.235982 ip-10-0-134-55 kubenswrapper[2564]: E0722 10:03:14.235959 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9d03ece0d116a98f58adfb8fd6cb20609c116c74f3555c6692ea7132425d1d\": container with ID starting with ee9d03ece0d116a98f58adfb8fd6cb20609c116c74f3555c6692ea7132425d1d not found: ID does not exist" containerID="ee9d03ece0d116a98f58adfb8fd6cb20609c116c74f3555c6692ea7132425d1d" Jul 22 10:03:14.236050 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.235992 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9d03ece0d116a98f58adfb8fd6cb20609c116c74f3555c6692ea7132425d1d"} err="failed to get container status \"ee9d03ece0d116a98f58adfb8fd6cb20609c116c74f3555c6692ea7132425d1d\": rpc error: code = NotFound desc = could not find container \"ee9d03ece0d116a98f58adfb8fd6cb20609c116c74f3555c6692ea7132425d1d\": container with ID starting with ee9d03ece0d116a98f58adfb8fd6cb20609c116c74f3555c6692ea7132425d1d not found: ID does not exist" Jul 22 10:03:14.251130 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.251106 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp"] Jul 22 10:03:14.255765 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.255745 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-99612b/rate-test-model-99612b-kserve-6884889567-q6kwp"] Jul 22 10:03:14.421392 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.421337 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" podUID="3c3bc65e-19f8-4205-bb54-fc8fc90d389e" containerName="main" containerID="cri-o://6ad966655e8ab1ba3b4853821773a040df74915877f8b9326f782a3cf0b43d58" gracePeriod=45 Jul 22 10:03:14.667912 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.667893 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:03:14.760729 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.760696 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cpxf\" (UniqueName: \"kubernetes.io/projected/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-kube-api-access-6cpxf\") pod \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " Jul 22 10:03:14.760890 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.760766 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-home\") pod \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " Jul 22 10:03:14.760890 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.760805 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-tmp-dir\") pod \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " Jul 22 10:03:14.760890 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.760827 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-model-cache\") pod \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " Jul 22 10:03:14.760890 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.760861 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-dshm\") pod \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " Jul 22 10:03:14.761098 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.760912 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-tls-certs\") pod \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\" (UID: \"3c3bc65e-19f8-4205-bb54-fc8fc90d389e\") " Jul 22 10:03:14.761157 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.761096 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-home" (OuterVolumeSpecName: "home") pod "3c3bc65e-19f8-4205-bb54-fc8fc90d389e" (UID: "3c3bc65e-19f8-4205-bb54-fc8fc90d389e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:03:14.761157 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.761105 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c3bc65e-19f8-4205-bb54-fc8fc90d389e" (UID: "3c3bc65e-19f8-4205-bb54-fc8fc90d389e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:03:14.761157 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.761116 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c3bc65e-19f8-4205-bb54-fc8fc90d389e" (UID: "3c3bc65e-19f8-4205-bb54-fc8fc90d389e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:03:14.761315 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.761283 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-home\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:03:14.761315 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.761299 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-tmp-dir\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:03:14.761388 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.761316 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-model-cache\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:03:14.762882 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.762856 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c3bc65e-19f8-4205-bb54-fc8fc90d389e" (UID: "3c3bc65e-19f8-4205-bb54-fc8fc90d389e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 10:03:14.762882 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.762872 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-dshm" (OuterVolumeSpecName: "dshm") pod "3c3bc65e-19f8-4205-bb54-fc8fc90d389e" (UID: "3c3bc65e-19f8-4205-bb54-fc8fc90d389e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 10:03:14.763076 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.762917 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-kube-api-access-6cpxf" (OuterVolumeSpecName: "kube-api-access-6cpxf") pod "3c3bc65e-19f8-4205-bb54-fc8fc90d389e" (UID: "3c3bc65e-19f8-4205-bb54-fc8fc90d389e"). InnerVolumeSpecName "kube-api-access-6cpxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 10:03:14.862260 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.862179 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-dshm\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:03:14.862260 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.862207 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-tls-certs\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:03:14.862260 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:14.862217 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cpxf\" (UniqueName: \"kubernetes.io/projected/3c3bc65e-19f8-4205-bb54-fc8fc90d389e-kube-api-access-6cpxf\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:03:15.234199 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.234160 2564 generic.go:356] "Generic (PLEG): container finished" podID="3c3bc65e-19f8-4205-bb54-fc8fc90d389e" containerID="6ad966655e8ab1ba3b4853821773a040df74915877f8b9326f782a3cf0b43d58" exitCode=0 Jul 22 10:03:15.234370 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.234205 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" Jul 22 10:03:15.234370 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.234250 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" event={"ID":"3c3bc65e-19f8-4205-bb54-fc8fc90d389e","Type":"ContainerDied","Data":"6ad966655e8ab1ba3b4853821773a040df74915877f8b9326f782a3cf0b43d58"} Jul 22 10:03:15.234370 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.234297 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m" event={"ID":"3c3bc65e-19f8-4205-bb54-fc8fc90d389e","Type":"ContainerDied","Data":"37107478cc6e8b27b7c7a0cb26c486d51fb12343bc4d184f5e0c3c07b08a45b1"} Jul 22 10:03:15.234370 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.234318 2564 scope.go:117] "RemoveContainer" containerID="6ad966655e8ab1ba3b4853821773a040df74915877f8b9326f782a3cf0b43d58" Jul 22 10:03:15.243128 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.243110 2564 scope.go:117] "RemoveContainer" containerID="6ad966655e8ab1ba3b4853821773a040df74915877f8b9326f782a3cf0b43d58" Jul 22 10:03:15.243376 ip-10-0-134-55 kubenswrapper[2564]: E0722 10:03:15.243358 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad966655e8ab1ba3b4853821773a040df74915877f8b9326f782a3cf0b43d58\": container with ID starting with 6ad966655e8ab1ba3b4853821773a040df74915877f8b9326f782a3cf0b43d58 not found: ID does not exist" containerID="6ad966655e8ab1ba3b4853821773a040df74915877f8b9326f782a3cf0b43d58" Jul 22 10:03:15.243422 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.243384 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad966655e8ab1ba3b4853821773a040df74915877f8b9326f782a3cf0b43d58"} err="failed to get container status \"6ad966655e8ab1ba3b4853821773a040df74915877f8b9326f782a3cf0b43d58\": rpc error: code = NotFound desc = could not find container \"6ad966655e8ab1ba3b4853821773a040df74915877f8b9326f782a3cf0b43d58\": container with ID starting with 6ad966655e8ab1ba3b4853821773a040df74915877f8b9326f782a3cf0b43d58 not found: ID does not exist" Jul 22 10:03:15.256262 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.256240 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m"] Jul 22 10:03:15.263173 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.263149 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-88ceff/rate-test-model-88ceff-kserve-6854d74f6c-7944m"] Jul 22 10:03:15.624377 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.624299 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7422-140659278542864"] Jul 22 10:03:15.624666 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.624654 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3bc65e-19f8-4205-bb54-fc8fc90d389e" containerName="main" Jul 22 10:03:15.624708 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.624667 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3bc65e-19f8-4205-bb54-fc8fc90d389e" containerName="main" Jul 22 10:03:15.624708 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.624685 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358f985d-b5a8-49f9-ae02-ef52b2fd21f8" containerName="main" Jul 22 10:03:15.624708 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.624692 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="358f985d-b5a8-49f9-ae02-ef52b2fd21f8" containerName="main" Jul 22 10:03:15.624796 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.624762 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="358f985d-b5a8-49f9-ae02-ef52b2fd21f8" containerName="main" Jul 22 10:03:15.624796 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.624768 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3bc65e-19f8-4205-bb54-fc8fc90d389e" containerName="main" Jul 22 10:03:15.629317 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.629294 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140659278542864" Jul 22 10:03:15.633815 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.632039 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dpq4v\"" Jul 22 10:03:15.634683 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.634658 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-140659278542864"] Jul 22 10:03:15.771066 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.771034 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25c8q\" (UniqueName: \"kubernetes.io/projected/8489e32e-4c3c-4eb8-913a-0162de223048-kube-api-access-25c8q\") pod \"test-curl-7422-140659278542864\" (UID: \"8489e32e-4c3c-4eb8-913a-0162de223048\") " pod="opendatahub/test-curl-7422-140659278542864" Jul 22 10:03:15.872028 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.871995 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25c8q\" (UniqueName: \"kubernetes.io/projected/8489e32e-4c3c-4eb8-913a-0162de223048-kube-api-access-25c8q\") pod \"test-curl-7422-140659278542864\" (UID: \"8489e32e-4c3c-4eb8-913a-0162de223048\") " pod="opendatahub/test-curl-7422-140659278542864" Jul 22 10:03:15.881211 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.881136 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25c8q\" (UniqueName: \"kubernetes.io/projected/8489e32e-4c3c-4eb8-913a-0162de223048-kube-api-access-25c8q\") pod \"test-curl-7422-140659278542864\" (UID: \"8489e32e-4c3c-4eb8-913a-0162de223048\") " pod="opendatahub/test-curl-7422-140659278542864" Jul 22 10:03:15.941449 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:15.941412 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140659278542864" Jul 22 10:03:16.014931 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:16.014894 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358f985d-b5a8-49f9-ae02-ef52b2fd21f8" path="/var/lib/kubelet/pods/358f985d-b5a8-49f9-ae02-ef52b2fd21f8/volumes" Jul 22 10:03:16.015371 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:16.015356 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c3bc65e-19f8-4205-bb54-fc8fc90d389e" path="/var/lib/kubelet/pods/3c3bc65e-19f8-4205-bb54-fc8fc90d389e/volumes" Jul 22 10:03:16.072460 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:16.072431 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-140659278542864"] Jul 22 10:03:16.074826 ip-10-0-134-55 kubenswrapper[2564]: W0722 10:03:16.074798 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8489e32e_4c3c_4eb8_913a_0162de223048.slice/crio-19c8509a29b18256faf0442095cf8df820208bdd8087217d43244e34cdf93ab6 WatchSource:0}: Error finding container 19c8509a29b18256faf0442095cf8df820208bdd8087217d43244e34cdf93ab6: Status 404 returned error can't find the container with id 19c8509a29b18256faf0442095cf8df820208bdd8087217d43244e34cdf93ab6 Jul 22 10:03:16.076220 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:16.076200 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 10:03:16.239881 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:16.239840 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-140659278542864" event={"ID":"8489e32e-4c3c-4eb8-913a-0162de223048","Type":"ContainerStarted","Data":"19c8509a29b18256faf0442095cf8df820208bdd8087217d43244e34cdf93ab6"} Jul 22 10:03:17.247165 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:17.247122 2564 generic.go:356] "Generic (PLEG): container finished" podID="8489e32e-4c3c-4eb8-913a-0162de223048" containerID="e756a3388e296c2a4253d74d00334debc8aaf0d6ddc4dc0ffb4223ebfba50ae5" exitCode=0 Jul 22 10:03:17.247539 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:17.247176 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-140659278542864" event={"ID":"8489e32e-4c3c-4eb8-913a-0162de223048","Type":"ContainerDied","Data":"e756a3388e296c2a4253d74d00334debc8aaf0d6ddc4dc0ffb4223ebfba50ae5"} Jul 22 10:03:18.381449 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.381424 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140659278542864" Jul 22 10:03:18.420402 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.420380 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7422-140659278542864_8489e32e-4c3c-4eb8-913a-0162de223048/test-curl-7422-140659278542864/0.log" Jul 22 10:03:18.444741 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.444712 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7422-140659278542864"] Jul 22 10:03:18.449083 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.449052 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7422-140659278542864"] Jul 22 10:03:18.495673 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.495654 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25c8q\" (UniqueName: \"kubernetes.io/projected/8489e32e-4c3c-4eb8-913a-0162de223048-kube-api-access-25c8q\") pod \"8489e32e-4c3c-4eb8-913a-0162de223048\" (UID: \"8489e32e-4c3c-4eb8-913a-0162de223048\") " Jul 22 10:03:18.497741 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.497713 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8489e32e-4c3c-4eb8-913a-0162de223048-kube-api-access-25c8q" (OuterVolumeSpecName: "kube-api-access-25c8q") pod "8489e32e-4c3c-4eb8-913a-0162de223048" (UID: "8489e32e-4c3c-4eb8-913a-0162de223048"). InnerVolumeSpecName "kube-api-access-25c8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 10:03:18.596518 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.596484 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25c8q\" (UniqueName: \"kubernetes.io/projected/8489e32e-4c3c-4eb8-913a-0162de223048-kube-api-access-25c8q\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:03:18.641395 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.641364 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99612b-rksdr"] Jul 22 10:03:18.641768 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.641754 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8489e32e-4c3c-4eb8-913a-0162de223048" containerName="test-curl-7422-140659278542864" Jul 22 10:03:18.641814 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.641769 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8489e32e-4c3c-4eb8-913a-0162de223048" containerName="test-curl-7422-140659278542864" Jul 22 10:03:18.641851 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.641827 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8489e32e-4c3c-4eb8-913a-0162de223048" containerName="test-curl-7422-140659278542864" Jul 22 10:03:18.644879 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.644862 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99612b-rksdr" Jul 22 10:03:18.647758 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.647740 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 10:03:18.647864 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.647808 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 10:03:18.655179 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.655158 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99612b-rksdr"] Jul 22 10:03:18.797824 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.797745 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2eb48dc-d685-476e-8514-e5939acc91cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-99612b-rksdr\" (UID: \"c2eb48dc-d685-476e-8514-e5939acc91cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99612b-rksdr" Jul 22 10:03:18.899139 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.899107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2eb48dc-d685-476e-8514-e5939acc91cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-99612b-rksdr\" (UID: \"c2eb48dc-d685-476e-8514-e5939acc91cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99612b-rksdr" Jul 22 10:03:18.899685 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.899667 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2eb48dc-d685-476e-8514-e5939acc91cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-99612b-rksdr\" (UID: \"c2eb48dc-d685-476e-8514-e5939acc91cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99612b-rksdr" Jul 22 10:03:18.955547 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:18.955508 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99612b-rksdr" Jul 22 10:03:19.082789 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:19.082767 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99612b-rksdr"] Jul 22 10:03:19.084839 ip-10-0-134-55 kubenswrapper[2564]: W0722 10:03:19.084805 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2eb48dc_d685_476e_8514_e5939acc91cc.slice/crio-cb17ebf55b519d25d597b01e6d411897550c547f226f8d1c70112c0e7b9dc761 WatchSource:0}: Error finding container cb17ebf55b519d25d597b01e6d411897550c547f226f8d1c70112c0e7b9dc761: Status 404 returned error can't find the container with id cb17ebf55b519d25d597b01e6d411897550c547f226f8d1c70112c0e7b9dc761 Jul 22 10:03:19.258262 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:19.258238 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-140659278542864" Jul 22 10:03:19.258419 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:19.258241 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19c8509a29b18256faf0442095cf8df820208bdd8087217d43244e34cdf93ab6" Jul 22 10:03:19.259419 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:19.259394 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99612b-rksdr" event={"ID":"c2eb48dc-d685-476e-8514-e5939acc91cc","Type":"ContainerStarted","Data":"cb17ebf55b519d25d597b01e6d411897550c547f226f8d1c70112c0e7b9dc761"} Jul 22 10:03:19.279828 ip-10-0-134-55 kubenswrapper[2564]: E0722 10:03:19.279801 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7422-140659278542864\" is forbidden: User \"system:node:ip-10-0-134-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-55.ec2.internal' and this object" podUID="8489e32e-4c3c-4eb8-913a-0162de223048" pod="opendatahub/test-curl-7422-140659278542864" Jul 22 10:03:20.015089 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:20.015054 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8489e32e-4c3c-4eb8-913a-0162de223048" path="/var/lib/kubelet/pods/8489e32e-4c3c-4eb8-913a-0162de223048/volumes" Jul 22 10:03:20.265344 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:20.265312 2564 generic.go:356] "Generic (PLEG): container finished" podID="c2eb48dc-d685-476e-8514-e5939acc91cc" containerID="0ddf44aa6a95aa6b8daa9830c013890220884c9d3fb9387ea1689573b3be1290" exitCode=0 Jul 22 10:03:20.265567 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:20.265388 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99612b-rksdr" event={"ID":"c2eb48dc-d685-476e-8514-e5939acc91cc","Type":"ContainerDied","Data":"0ddf44aa6a95aa6b8daa9830c013890220884c9d3fb9387ea1689573b3be1290"} Jul 22 10:03:21.402407 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:21.402383 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99612b-rksdr" Jul 22 10:03:21.524868 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:21.524781 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2eb48dc-d685-476e-8514-e5939acc91cc-maas-api-service-ca\") pod \"c2eb48dc-d685-476e-8514-e5939acc91cc\" (UID: \"c2eb48dc-d685-476e-8514-e5939acc91cc\") " Jul 22 10:03:21.525217 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:21.525189 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2eb48dc-d685-476e-8514-e5939acc91cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2eb48dc-d685-476e-8514-e5939acc91cc" (UID: "c2eb48dc-d685-476e-8514-e5939acc91cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 10:03:21.625607 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:21.625576 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2eb48dc-d685-476e-8514-e5939acc91cc-maas-api-service-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:03:22.276661 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:22.276573 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99612b-rksdr" event={"ID":"c2eb48dc-d685-476e-8514-e5939acc91cc","Type":"ContainerDied","Data":"cb17ebf55b519d25d597b01e6d411897550c547f226f8d1c70112c0e7b9dc761"} Jul 22 10:03:22.276661 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:22.276596 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99612b-rksdr" Jul 22 10:03:22.276661 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:22.276625 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb17ebf55b519d25d597b01e6d411897550c547f226f8d1c70112c0e7b9dc761" Jul 22 10:03:29.242927 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:29.242889 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99612b-29745240-d7tg8"] Jul 22 10:03:29.249389 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:29.249361 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-99612b-29745240-d7tg8"] Jul 22 10:03:30.014655 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:30.014617 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18ee02aa-2d47-4e48-a95a-e279a150444f" path="/var/lib/kubelet/pods/18ee02aa-2d47-4e48-a95a-e279a150444f/volumes" Jul 22 10:03:32.358074 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:32.358043 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 10:03:33.732270 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:33.732234 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99612b-rksdr"] Jul 22 10:03:33.734732 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:33.734706 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-99612b-rksdr"] Jul 22 10:03:34.014719 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:34.014640 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2eb48dc-d685-476e-8514-e5939acc91cc" path="/var/lib/kubelet/pods/c2eb48dc-d685-476e-8514-e5939acc91cc/volumes" Jul 22 10:03:41.073448 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:41.073412 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj"] Jul 22 10:03:41.073951 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:41.073820 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2eb48dc-d685-476e-8514-e5939acc91cc" containerName="revoke-keys" Jul 22 10:03:41.073951 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:41.073832 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2eb48dc-d685-476e-8514-e5939acc91cc" containerName="revoke-keys" Jul 22 10:03:41.073951 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:41.073910 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2eb48dc-d685-476e-8514-e5939acc91cc" containerName="revoke-keys" Jul 22 10:03:41.076865 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:41.076839 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj" Jul 22 10:03:41.079378 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:41.079359 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 10:03:41.079556 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:41.079527 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tmn2j\"" Jul 22 10:03:41.088521 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:41.088498 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj"] Jul 22 10:03:41.113072 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:41.113040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df0283b-4ddd-4258-9acf-0fc927cad4fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj\" (UID: \"0df0283b-4ddd-4258-9acf-0fc927cad4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj" Jul 22 10:03:41.214506 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:41.214460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df0283b-4ddd-4258-9acf-0fc927cad4fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj\" (UID: \"0df0283b-4ddd-4258-9acf-0fc927cad4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj" Jul 22 10:03:41.215139 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:41.215121 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df0283b-4ddd-4258-9acf-0fc927cad4fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj\" (UID: \"0df0283b-4ddd-4258-9acf-0fc927cad4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj" Jul 22 10:03:41.387191 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:41.387103 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj" Jul 22 10:03:41.524279 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:41.524257 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj"] Jul 22 10:03:41.526185 ip-10-0-134-55 kubenswrapper[2564]: W0722 10:03:41.526158 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df0283b_4ddd_4258_9acf_0fc927cad4fa.slice/crio-7a43c2c3450ec695bc78610434793fc3bda397ed1a4d60b30aaf7427a293e241 WatchSource:0}: Error finding container 7a43c2c3450ec695bc78610434793fc3bda397ed1a4d60b30aaf7427a293e241: Status 404 returned error can't find the container with id 7a43c2c3450ec695bc78610434793fc3bda397ed1a4d60b30aaf7427a293e241 Jul 22 10:03:42.378187 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:42.378153 2564 generic.go:356] "Generic (PLEG): container finished" podID="0df0283b-4ddd-4258-9acf-0fc927cad4fa" containerID="9e6b030c2121cc39c995ec31750dcfa47ff4fb99435bb0b4028d2cecc79cff75" exitCode=0 Jul 22 10:03:42.378580 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:42.378194 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj" event={"ID":"0df0283b-4ddd-4258-9acf-0fc927cad4fa","Type":"ContainerDied","Data":"9e6b030c2121cc39c995ec31750dcfa47ff4fb99435bb0b4028d2cecc79cff75"} Jul 22 10:03:42.378580 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:42.378233 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj" event={"ID":"0df0283b-4ddd-4258-9acf-0fc927cad4fa","Type":"ContainerStarted","Data":"7a43c2c3450ec695bc78610434793fc3bda397ed1a4d60b30aaf7427a293e241"} Jul 22 10:03:43.512368 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:43.512345 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj" Jul 22 10:03:43.636921 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:43.636825 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df0283b-4ddd-4258-9acf-0fc927cad4fa-maas-api-service-ca\") pod \"0df0283b-4ddd-4258-9acf-0fc927cad4fa\" (UID: \"0df0283b-4ddd-4258-9acf-0fc927cad4fa\") " Jul 22 10:03:43.637269 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:43.637238 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df0283b-4ddd-4258-9acf-0fc927cad4fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0df0283b-4ddd-4258-9acf-0fc927cad4fa" (UID: "0df0283b-4ddd-4258-9acf-0fc927cad4fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 10:03:43.737853 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:43.737818 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df0283b-4ddd-4258-9acf-0fc927cad4fa-maas-api-service-ca\") on node \"ip-10-0-134-55.ec2.internal\" DevicePath \"\"" Jul 22 10:03:44.388678 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:44.388591 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj" Jul 22 10:03:44.388678 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:44.388599 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj" event={"ID":"0df0283b-4ddd-4258-9acf-0fc927cad4fa","Type":"ContainerDied","Data":"7a43c2c3450ec695bc78610434793fc3bda397ed1a4d60b30aaf7427a293e241"} Jul 22 10:03:44.388678 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:44.388640 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a43c2c3450ec695bc78610434793fc3bda397ed1a4d60b30aaf7427a293e241" Jul 22 10:03:51.590842 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:51.590805 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-88ceff-29745240-5mwvs"] Jul 22 10:03:51.593832 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:51.593807 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-88ceff-29745240-5mwvs"] Jul 22 10:03:52.014301 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:52.014264 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26368474-2cec-4b99-b369-7497b52015b4" path="/var/lib/kubelet/pods/26368474-2cec-4b99-b369-7497b52015b4/volumes" Jul 22 10:03:54.252827 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:54.252792 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5488w"] Jul 22 10:03:56.168078 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:56.168045 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj"] Jul 22 10:03:56.170174 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:56.170152 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-88ceff-kq5lj"] Jul 22 10:03:58.014090 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:03:58.014053 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0df0283b-4ddd-4258-9acf-0fc927cad4fa" path="/var/lib/kubelet/pods/0df0283b-4ddd-4258-9acf-0fc927cad4fa/volumes" Jul 22 10:04:09.908986 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:09.908930 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67797c6b-jwjxz_b9ca8253-c53a-4a53-82fb-ef777b42a7e8/copy-manifests/0.log" Jul 22 10:04:09.915832 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:09.915805 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67797c6b-jwjxz_b9ca8253-c53a-4a53-82fb-ef777b42a7e8/manager/0.log" Jul 22 10:04:10.900603 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:10.900570 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29745240-wsbb8_04284088-dea8-46e3-bd1b-8633c768f845/cleanup/0.log" Jul 22 10:04:12.727880 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:12.727851 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-v4pxw_c1d40468-de6d-43e6-9257-b448ca3fc792/kuadrant-console-plugin/0.log" Jul 22 10:04:13.081356 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:13.081266 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-5488w_05214e24-d32c-403f-9bbd-73535684630e/limitador/0.log" Jul 22 10:04:13.902861 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:13.902835 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68755c4d9c-rcxzb_42b405e7-01fd-4676-837b-ade2217d49cc/kube-auth-proxy/0.log" Jul 22 10:04:15.108921 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:15.108892 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8mlnj_c1ad39d9-6992-4fb6-a401-098c6bfa63f8/main/0.log" Jul 22 10:04:15.339053 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:15.339019 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-nkz2n_45f6ea2f-a25e-44f9-b357-2495db4265fe/main/0.log" Jul 22 10:04:20.365788 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.365710 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nzwpt/must-gather-kx4j7"] Jul 22 10:04:20.366182 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.366096 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df0283b-4ddd-4258-9acf-0fc927cad4fa" containerName="revoke-keys" Jul 22 10:04:20.366182 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.366107 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df0283b-4ddd-4258-9acf-0fc927cad4fa" containerName="revoke-keys" Jul 22 10:04:20.366182 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.366172 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0df0283b-4ddd-4258-9acf-0fc927cad4fa" containerName="revoke-keys" Jul 22 10:04:20.369504 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.369486 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzwpt/must-gather-kx4j7" Jul 22 10:04:20.372013 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.371972 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nzwpt\"/\"default-dockercfg-54vb5\"" Jul 22 10:04:20.372136 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.372032 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nzwpt\"/\"openshift-service-ca.crt\"" Jul 22 10:04:20.372259 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.372245 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nzwpt\"/\"kube-root-ca.crt\"" Jul 22 10:04:20.383394 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.383369 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzwpt/must-gather-kx4j7"] Jul 22 10:04:20.484473 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.484439 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds8p7\" (UniqueName: \"kubernetes.io/projected/f1a90511-49f1-40e1-bddf-2797c317783c-kube-api-access-ds8p7\") pod \"must-gather-kx4j7\" (UID: \"f1a90511-49f1-40e1-bddf-2797c317783c\") " pod="openshift-must-gather-nzwpt/must-gather-kx4j7" Jul 22 10:04:20.484643 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.484495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f1a90511-49f1-40e1-bddf-2797c317783c-must-gather-output\") pod \"must-gather-kx4j7\" (UID: \"f1a90511-49f1-40e1-bddf-2797c317783c\") " pod="openshift-must-gather-nzwpt/must-gather-kx4j7" Jul 22 10:04:20.585322 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.585295 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds8p7\" (UniqueName: \"kubernetes.io/projected/f1a90511-49f1-40e1-bddf-2797c317783c-kube-api-access-ds8p7\") pod \"must-gather-kx4j7\" (UID: \"f1a90511-49f1-40e1-bddf-2797c317783c\") " pod="openshift-must-gather-nzwpt/must-gather-kx4j7" Jul 22 10:04:20.585514 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.585353 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f1a90511-49f1-40e1-bddf-2797c317783c-must-gather-output\") pod \"must-gather-kx4j7\" (UID: \"f1a90511-49f1-40e1-bddf-2797c317783c\") " pod="openshift-must-gather-nzwpt/must-gather-kx4j7" Jul 22 10:04:20.585784 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.585762 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f1a90511-49f1-40e1-bddf-2797c317783c-must-gather-output\") pod \"must-gather-kx4j7\" (UID: \"f1a90511-49f1-40e1-bddf-2797c317783c\") " pod="openshift-must-gather-nzwpt/must-gather-kx4j7" Jul 22 10:04:20.593811 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.593792 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds8p7\" (UniqueName: \"kubernetes.io/projected/f1a90511-49f1-40e1-bddf-2797c317783c-kube-api-access-ds8p7\") pod \"must-gather-kx4j7\" (UID: \"f1a90511-49f1-40e1-bddf-2797c317783c\") " pod="openshift-must-gather-nzwpt/must-gather-kx4j7" Jul 22 10:04:20.678073 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.678002 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzwpt/must-gather-kx4j7" Jul 22 10:04:20.817740 ip-10-0-134-55 kubenswrapper[2564]: W0722 10:04:20.817712 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1a90511_49f1_40e1_bddf_2797c317783c.slice/crio-c4e84b1c009fc8fb4214db2b2c9809366bbb9e1048850df02228c149c9363180 WatchSource:0}: Error finding container c4e84b1c009fc8fb4214db2b2c9809366bbb9e1048850df02228c149c9363180: Status 404 returned error can't find the container with id c4e84b1c009fc8fb4214db2b2c9809366bbb9e1048850df02228c149c9363180 Jul 22 10:04:20.821384 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:20.821359 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzwpt/must-gather-kx4j7"] Jul 22 10:04:21.576133 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:21.576093 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzwpt/must-gather-kx4j7" event={"ID":"f1a90511-49f1-40e1-bddf-2797c317783c","Type":"ContainerStarted","Data":"c4e84b1c009fc8fb4214db2b2c9809366bbb9e1048850df02228c149c9363180"} Jul 22 10:04:22.585371 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:22.585327 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzwpt/must-gather-kx4j7" event={"ID":"f1a90511-49f1-40e1-bddf-2797c317783c","Type":"ContainerStarted","Data":"cd1f64fa4b707b7b319128f6fb3d96f65e1683bbcfed0a7a59992cb8855fa2d4"} Jul 22 10:04:22.585865 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:22.585388 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzwpt/must-gather-kx4j7" event={"ID":"f1a90511-49f1-40e1-bddf-2797c317783c","Type":"ContainerStarted","Data":"0ab48dc38dd7266105c4034c6354b49fd6987cbef898ae145538a39aa4b68178"} Jul 22 10:04:22.603406 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:22.603357 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nzwpt/must-gather-kx4j7" podStartSLOduration=1.792342082 podStartE2EDuration="2.603342877s" totalImagesPullingTime="811.000795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 10:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 10:04:22.601531144 +0000 UTC m=+2787.099038651" watchObservedRunningTime="2026-07-22 10:04:22.603342877 +0000 UTC m=+2787.100850291" Jul 22 10:04:23.354158 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:23.354116 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5zw6v_21a0247b-c2c6-4f3e-a619-2eaaf17a2a06/global-pull-secret-syncer/0.log" Jul 22 10:04:23.479325 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:23.479278 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xqb5q_9425446d-1c7a-419d-8020-59828616976a/konnectivity-agent/0.log" Jul 22 10:04:23.505122 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:23.505090 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-55.ec2.internal_fceedcd76d463c4089e6465fad28b1d6/haproxy/0.log" Jul 22 10:04:28.950083 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:28.949996 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-v4pxw_c1d40468-de6d-43e6-9257-b448ca3fc792/kuadrant-console-plugin/0.log" Jul 22 10:04:29.228875 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:29.228794 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-5488w_05214e24-d32c-403f-9bbd-73535684630e/limitador/0.log" Jul 22 10:04:30.951690 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:30.951591 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9519484e-2793-4641-8937-82d24bed53b3/alertmanager/0.log" Jul 22 10:04:30.976428 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:30.976385 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9519484e-2793-4641-8937-82d24bed53b3/config-reloader/0.log" Jul 22 10:04:31.003254 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.003222 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9519484e-2793-4641-8937-82d24bed53b3/kube-rbac-proxy-web/0.log" Jul 22 10:04:31.030341 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.030302 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9519484e-2793-4641-8937-82d24bed53b3/kube-rbac-proxy/0.log" Jul 22 10:04:31.057785 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.057707 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9519484e-2793-4641-8937-82d24bed53b3/kube-rbac-proxy-metric/0.log" Jul 22 10:04:31.086225 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.086127 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9519484e-2793-4641-8937-82d24bed53b3/prom-label-proxy/0.log" Jul 22 10:04:31.114708 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.114662 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9519484e-2793-4641-8937-82d24bed53b3/init-config-reloader/0.log" Jul 22 10:04:31.188095 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.188029 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-xbttw_2ad4272f-1391-4472-a944-c3bc4f8ddfb5/kube-state-metrics/0.log" Jul 22 10:04:31.209768 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.209663 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-xbttw_2ad4272f-1391-4472-a944-c3bc4f8ddfb5/kube-rbac-proxy-main/0.log" Jul 22 10:04:31.234485 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.234446 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-xbttw_2ad4272f-1391-4472-a944-c3bc4f8ddfb5/kube-rbac-proxy-self/0.log" Jul 22 10:04:31.264959 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.264901 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7fbb65654c-7q7l5_f0832a10-9283-40e6-bb8e-24b6dd30f4e7/metrics-server/0.log" Jul 22 10:04:31.294459 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.294429 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-tqtct_b093f3a3-f6e3-4692-be96-8319e49aea36/monitoring-plugin/0.log" Jul 22 10:04:31.341564 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.341520 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fc79w_9730e0f1-91ed-44fa-ae9f-df4e8ed99a15/node-exporter/0.log" Jul 22 10:04:31.368875 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.368848 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fc79w_9730e0f1-91ed-44fa-ae9f-df4e8ed99a15/kube-rbac-proxy/0.log" Jul 22 10:04:31.402860 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.402830 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fc79w_9730e0f1-91ed-44fa-ae9f-df4e8ed99a15/init-textfile/0.log" Jul 22 10:04:31.604739 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.604661 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-rbpds_21d042ab-2267-4459-9215-01e50bf7eee8/kube-rbac-proxy-main/0.log" Jul 22 10:04:31.628857 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.628829 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-rbpds_21d042ab-2267-4459-9215-01e50bf7eee8/kube-rbac-proxy-self/0.log" Jul 22 10:04:31.654082 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.654054 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-rbpds_21d042ab-2267-4459-9215-01e50bf7eee8/openshift-state-metrics/0.log" Jul 22 10:04:31.689732 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.689702 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4fa27dbe-6518-441c-bdcb-284d2377fce2/prometheus/0.log" Jul 22 10:04:31.713194 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.713163 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4fa27dbe-6518-441c-bdcb-284d2377fce2/config-reloader/0.log" Jul 22 10:04:31.742186 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.742155 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4fa27dbe-6518-441c-bdcb-284d2377fce2/thanos-sidecar/0.log" Jul 22 10:04:31.769663 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.769639 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4fa27dbe-6518-441c-bdcb-284d2377fce2/kube-rbac-proxy-web/0.log" Jul 22 10:04:31.796878 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.796766 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4fa27dbe-6518-441c-bdcb-284d2377fce2/kube-rbac-proxy/0.log" Jul 22 10:04:31.820469 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.820439 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4fa27dbe-6518-441c-bdcb-284d2377fce2/kube-rbac-proxy-thanos/0.log" Jul 22 10:04:31.843827 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.843803 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4fa27dbe-6518-441c-bdcb-284d2377fce2/init-config-reloader/0.log" Jul 22 10:04:31.874270 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.874188 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-6zrzx_3a5806ce-e8e1-4a25-8bfc-15f559539bb1/prometheus-operator/0.log" Jul 22 10:04:31.896666 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.896635 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-6zrzx_3a5806ce-e8e1-4a25-8bfc-15f559539bb1/kube-rbac-proxy/0.log" Jul 22 10:04:31.924013 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.923986 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-4hmnr_73cdba8d-82d0-4c79-a4f1-dea7c3fa8036/prometheus-operator-admission-webhook/0.log" Jul 22 10:04:31.959413 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.959360 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59b5558bb7-dvc2r_628bdc9e-cc5a-41e9-be17-520d10f0a4f6/telemeter-client/0.log" Jul 22 10:04:31.982837 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:31.982806 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59b5558bb7-dvc2r_628bdc9e-cc5a-41e9-be17-520d10f0a4f6/reload/0.log" Jul 22 10:04:32.013317 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:32.013287 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59b5558bb7-dvc2r_628bdc9e-cc5a-41e9-be17-520d10f0a4f6/kube-rbac-proxy/0.log" Jul 22 10:04:32.044905 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:32.044867 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66c7958d54-c7w2d_3dae0afa-7c17-4b64-8f0f-fbbf46c632da/thanos-query/0.log" Jul 22 10:04:32.069656 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:32.069618 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66c7958d54-c7w2d_3dae0afa-7c17-4b64-8f0f-fbbf46c632da/kube-rbac-proxy-web/0.log" Jul 22 10:04:32.093244 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:32.093211 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66c7958d54-c7w2d_3dae0afa-7c17-4b64-8f0f-fbbf46c632da/kube-rbac-proxy/0.log" Jul 22 10:04:32.118270 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:32.118239 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66c7958d54-c7w2d_3dae0afa-7c17-4b64-8f0f-fbbf46c632da/prom-label-proxy/0.log" Jul 22 10:04:32.141926 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:32.141852 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66c7958d54-c7w2d_3dae0afa-7c17-4b64-8f0f-fbbf46c632da/kube-rbac-proxy-rules/0.log" Jul 22 10:04:32.167294 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:32.167268 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-66c7958d54-c7w2d_3dae0afa-7c17-4b64-8f0f-fbbf46c632da/kube-rbac-proxy-metrics/0.log" Jul 22 10:04:33.220486 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.220448 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59"] Jul 22 10:04:33.227569 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.227538 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.238096 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.238031 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59"] Jul 22 10:04:33.335600 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.335545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6960158a-29f5-44b3-ae84-67fc9f937d48-podres\") pod \"perf-node-gather-daemonset-52q59\" (UID: \"6960158a-29f5-44b3-ae84-67fc9f937d48\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.335801 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.335621 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xqbk\" (UniqueName: \"kubernetes.io/projected/6960158a-29f5-44b3-ae84-67fc9f937d48-kube-api-access-7xqbk\") pod \"perf-node-gather-daemonset-52q59\" (UID: \"6960158a-29f5-44b3-ae84-67fc9f937d48\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.335801 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.335707 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6960158a-29f5-44b3-ae84-67fc9f937d48-proc\") pod \"perf-node-gather-daemonset-52q59\" (UID: \"6960158a-29f5-44b3-ae84-67fc9f937d48\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.335801 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.335738 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6960158a-29f5-44b3-ae84-67fc9f937d48-lib-modules\") pod \"perf-node-gather-daemonset-52q59\" (UID: \"6960158a-29f5-44b3-ae84-67fc9f937d48\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.335801 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.335792 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6960158a-29f5-44b3-ae84-67fc9f937d48-sys\") pod \"perf-node-gather-daemonset-52q59\" (UID: \"6960158a-29f5-44b3-ae84-67fc9f937d48\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.436645 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.436595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xqbk\" (UniqueName: \"kubernetes.io/projected/6960158a-29f5-44b3-ae84-67fc9f937d48-kube-api-access-7xqbk\") pod \"perf-node-gather-daemonset-52q59\" (UID: \"6960158a-29f5-44b3-ae84-67fc9f937d48\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.436833 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.436703 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6960158a-29f5-44b3-ae84-67fc9f937d48-proc\") pod \"perf-node-gather-daemonset-52q59\" (UID: \"6960158a-29f5-44b3-ae84-67fc9f937d48\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.436833 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.436746 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6960158a-29f5-44b3-ae84-67fc9f937d48-lib-modules\") pod \"perf-node-gather-daemonset-52q59\" (UID: \"6960158a-29f5-44b3-ae84-67fc9f937d48\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.436833 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.436803 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6960158a-29f5-44b3-ae84-67fc9f937d48-sys\") pod \"perf-node-gather-daemonset-52q59\" (UID: \"6960158a-29f5-44b3-ae84-67fc9f937d48\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.436972 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.436865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6960158a-29f5-44b3-ae84-67fc9f937d48-podres\") pod \"perf-node-gather-daemonset-52q59\" (UID: \"6960158a-29f5-44b3-ae84-67fc9f937d48\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.437087 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.437068 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6960158a-29f5-44b3-ae84-67fc9f937d48-podres\") pod \"perf-node-gather-daemonset-52q59\" (UID: \"6960158a-29f5-44b3-ae84-67fc9f937d48\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.437155 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.437129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6960158a-29f5-44b3-ae84-67fc9f937d48-sys\") pod \"perf-node-gather-daemonset-52q59\" (UID: \"6960158a-29f5-44b3-ae84-67fc9f937d48\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.437206 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.437172 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6960158a-29f5-44b3-ae84-67fc9f937d48-proc\") pod \"perf-node-gather-daemonset-52q59\" (UID: \"6960158a-29f5-44b3-ae84-67fc9f937d48\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.437206 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.437174 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6960158a-29f5-44b3-ae84-67fc9f937d48-lib-modules\") pod \"perf-node-gather-daemonset-52q59\" (UID: \"6960158a-29f5-44b3-ae84-67fc9f937d48\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.455130 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.445173 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xqbk\" (UniqueName: \"kubernetes.io/projected/6960158a-29f5-44b3-ae84-67fc9f937d48-kube-api-access-7xqbk\") pod \"perf-node-gather-daemonset-52q59\" (UID: \"6960158a-29f5-44b3-ae84-67fc9f937d48\") " pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.543288 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.543189 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:33.777779 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:33.777755 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59"] Jul 22 10:04:34.566258 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:34.566231 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b966bd857-skg5r_8939cc7a-a481-4a67-8836-92edc5e15f9a/console/0.log" Jul 22 10:04:34.695777 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:34.695741 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" event={"ID":"6960158a-29f5-44b3-ae84-67fc9f937d48","Type":"ContainerStarted","Data":"1bb58629090f5d3cfab0cd1ff75654611c2f4acf5285eb1f57f59890b179c6fa"} Jul 22 10:04:34.696066 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:34.696042 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" event={"ID":"6960158a-29f5-44b3-ae84-67fc9f937d48","Type":"ContainerStarted","Data":"d2ea75b1d1cd39481fe615f5a335d254053683ecef7a5fdf5a7aadfc18efc2e3"} Jul 22 10:04:34.696248 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:34.696227 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:34.713988 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:34.713901 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" podStartSLOduration=1.713885334 podStartE2EDuration="1.713885334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 10:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 10:04:34.712169732 +0000 UTC m=+2799.209677176" watchObservedRunningTime="2026-07-22 10:04:34.713885334 +0000 UTC m=+2799.211392770" Jul 22 10:04:36.331538 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:36.331507 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pkdz2_6106e154-c174-4852-a1c0-52d15d988868/dns/0.log" Jul 22 10:04:36.355897 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:36.355868 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pkdz2_6106e154-c174-4852-a1c0-52d15d988868/kube-rbac-proxy/0.log" Jul 22 10:04:36.467205 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:36.467178 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p9fv9_94e33bbc-9879-4500-abd0-a3cde0baf87f/dns-node-resolver/0.log" Jul 22 10:04:37.078311 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:37.078275 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-98sq4_9d4fc604-5bcb-4274-a20d-5710d075dce3/node-ca/0.log" Jul 22 10:04:38.280387 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:38.280356 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68755c4d9c-rcxzb_42b405e7-01fd-4676-837b-ade2217d49cc/kube-auth-proxy/0.log" Jul 22 10:04:39.187349 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:39.187311 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sgnqf_cb2ebe90-04ef-463b-a79d-374233cb36dd/serve-healthcheck-canary/0.log" Jul 22 10:04:39.766648 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:39.766621 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hntwv_800290ab-f0d6-4e93-82a8-5eea0b266100/kube-rbac-proxy/0.log" Jul 22 10:04:39.793559 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:39.793523 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hntwv_800290ab-f0d6-4e93-82a8-5eea0b266100/exporter/0.log" Jul 22 10:04:39.816525 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:39.816496 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hntwv_800290ab-f0d6-4e93-82a8-5eea0b266100/extractor/0.log" Jul 22 10:04:40.712964 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:40.712922 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nzwpt/perf-node-gather-daemonset-52q59" Jul 22 10:04:42.656153 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:42.656122 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67797c6b-jwjxz_b9ca8253-c53a-4a53-82fb-ef777b42a7e8/manager/0.log" Jul 22 10:04:42.682668 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:42.682643 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67797c6b-jwjxz_b9ca8253-c53a-4a53-82fb-ef777b42a7e8/copy-manifests/0.log" Jul 22 10:04:44.149467 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:44.149438 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7d9cd798d9-csxx2_cd6bdb8c-4a7c-4a86-8a75-d284f666fa4a/manager/0.log" Jul 22 10:04:44.208387 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:44.208352 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-29k7l_465cffd0-e0b5-47cb-a8cd-0e1962c030ab/openshift-lws-operator/0.log" Jul 22 10:04:44.226750 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:44.226716 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68755c4d9c-rcxzb_42b405e7-01fd-4676-837b-ade2217d49cc/kube-auth-proxy/0.log" Jul 22 10:04:45.684062 ip-10-0-134-55 kubenswrapper[2564]: I0722 10:04:45.684037 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68755c4d9c-rcxzb_42b405e7-01fd-4676-837b-ade2217d49cc/kube-auth-proxy/0.log"