Jul 29 08:46:04.047589 ip-10-0-141-66 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 29 08:46:04.047601 ip-10-0-141-66 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 29 08:46:04.047608 ip-10-0-141-66 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 29 08:46:04.047830 ip-10-0-141-66 systemd[1]: Failed to start Kubernetes Kubelet. Jul 29 08:46:15.323307 ip-10-0-141-66 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 29 08:46:15.323324 ip-10-0-141-66 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8f6925cd3f0e469292bea918163c616b -- Jul 29 08:48:06.775647 ip-10-0-141-66 systemd[1]: Starting Kubernetes Kubelet... Jul 29 08:48:07.110280 ip-10-0-141-66 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 08:48:07.110280 ip-10-0-141-66 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 08:48:07.110280 ip-10-0-141-66 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 08:48:07.110280 ip-10-0-141-66 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 08:48:07.110280 ip-10-0-141-66 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 08:48:07.112323 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.112225 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 08:48:07.118716 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118690 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:48:07.118716 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118712 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:48:07.118716 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118716 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:48:07.118716 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118720 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:48:07.118716 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118723 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:48:07.118716 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118726 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118729 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118734 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118737 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118740 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118744 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118749 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118753 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118756 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118758 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118761 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118764 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118767 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118770 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118773 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118780 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118783 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118786 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118789 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118791 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:48:07.118982 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118794 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118796 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118799 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118801 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118804 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118807 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118810 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118814 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118818 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118821 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118824 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118827 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118829 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118832 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118835 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118838 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118843 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118848 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118850 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:48:07.119488 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118853 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118856 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118858 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118861 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118863 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118866 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118869 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118871 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118874 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118877 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118879 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118883 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118886 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118888 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118891 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118893 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118896 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118898 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118902 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118904 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:48:07.119963 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118907 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118910 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118913 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118915 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118919 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118921 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118925 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118928 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118931 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118933 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118936 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118939 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118941 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118944 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118948 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118950 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118953 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118955 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118958 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118961 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:48:07.120469 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118964 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118967 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118969 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118972 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118974 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118977 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118979 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118982 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118985 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118988 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118990 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118993 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118996 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.118999 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119001 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119755 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119762 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119765 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119768 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119771 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:48:07.120945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119774 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119778 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119781 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119783 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119786 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119789 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119791 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119794 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119798 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119802 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119806 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119809 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119812 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119816 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119819 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119821 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119824 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119827 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119829 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119832 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:48:07.121434 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119835 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119838 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119841 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119844 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119847 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119849 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119852 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119854 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119857 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119860 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119862 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119868 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119870 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119873 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119876 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119878 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119882 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119884 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119887 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119890 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:48:07.121927 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119892 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119895 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119897 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119900 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119902 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119906 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119909 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119911 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119914 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119917 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119920 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119922 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119925 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119927 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119931 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119933 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119936 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119939 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119942 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119945 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:48:07.122432 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119948 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119950 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119953 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119957 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119960 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119963 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119965 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119968 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119971 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119973 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119976 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119979 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119981 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119984 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119986 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119989 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119991 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119994 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.119997 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:48:07.122945 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120000 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120002 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120006 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120009 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120012 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120016 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120019 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120022 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120025 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120028 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120030 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120033 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120036 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120039 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120042 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120447 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120455 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120458 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120462 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:48:07.123437 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120465 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120468 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120471 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120474 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120476 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120479 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120482 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120485 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120487 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120490 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120493 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120495 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120499 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120502 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120504 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120507 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120509 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120512 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120514 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120517 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:48:07.123910 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120520 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120522 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120525 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120527 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120530 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120533 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120535 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120538 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120540 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120544 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120547 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120550 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120552 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120555 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120558 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120560 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120563 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120565 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120568 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120570 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:48:07.124450 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120573 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120576 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120579 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120581 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120584 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120587 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120590 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120592 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120594 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120597 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120599 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120602 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120605 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120607 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120610 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120612 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120615 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120619 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120624 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:48:07.124941 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120628 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120631 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120635 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120638 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120641 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120644 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120647 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120649 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120654 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120657 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120659 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120662 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120664 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120667 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120670 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120672 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120675 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120677 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120680 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120683 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:48:07.125440 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120685 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120688 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120690 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120693 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120696 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120698 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120701 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120703 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120705 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120709 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120713 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120717 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120720 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120723 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120727 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.120730 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120802 2563 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120810 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120817 2563 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120821 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 08:48:07.125952 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120825 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120829 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120833 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120837 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120841 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120844 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120847 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120850 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120854 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120857 2563 flags.go:64] FLAG: --cgroup-root="" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120860 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120862 2563 flags.go:64] FLAG: --client-ca-file="" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120868 2563 flags.go:64] FLAG: --cloud-provider="external" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120873 2563 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120891 2563 flags.go:64] FLAG: --cluster-domain="" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120896 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120900 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120904 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120907 2563 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120911 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120914 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120917 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120921 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120925 2563 flags.go:64] FLAG: --contention-profiling="false" Jul 29 08:48:07.126457 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120928 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120931 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120936 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120939 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120943 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120946 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120949 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120952 2563 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120955 2563 flags.go:64] FLAG: --enable-server="true" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120959 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120967 2563 flags.go:64] FLAG: --event-burst="100" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120970 2563 flags.go:64] FLAG: --event-qps="50" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120973 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120976 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120979 2563 flags.go:64] FLAG: --eviction-hard="" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120982 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120985 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120989 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120992 2563 flags.go:64] FLAG: --eviction-soft="" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120994 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.120997 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121007 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121010 2563 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121013 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121016 2563 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 08:48:07.127246 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121019 2563 flags.go:64] FLAG: --feature-gates="" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121023 2563 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121026 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121029 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121033 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121036 2563 flags.go:64] FLAG: --healthz-port="10248" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121040 2563 flags.go:64] FLAG: --help="false" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121043 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121046 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121050 2563 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121053 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121056 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121059 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121062 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121065 2563 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121068 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121071 2563 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121074 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121077 2563 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121080 2563 flags.go:64] FLAG: --kube-reserved="" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121083 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121086 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121089 2563 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121091 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 08:48:07.128349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121094 2563 flags.go:64] FLAG: --lock-file="" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121097 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121100 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121103 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121108 2563 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121112 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121115 2563 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121118 2563 flags.go:64] FLAG: --logging-format="text" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121121 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121124 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121127 2563 flags.go:64] FLAG: --manifest-url="" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121130 2563 flags.go:64] FLAG: --manifest-url-header="" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121138 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121141 2563 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121146 2563 flags.go:64] FLAG: --max-pods="110" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121149 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121152 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121155 2563 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121158 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121161 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121165 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121168 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121176 2563 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121179 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121183 2563 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 08:48:07.129388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121186 2563 flags.go:64] FLAG: --pod-cidr="" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121189 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121194 2563 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121197 2563 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121200 2563 flags.go:64] FLAG: --pods-per-core="0" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121203 2563 flags.go:64] FLAG: --port="10250" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121206 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121209 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0074addff70bdfce6" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121212 2563 flags.go:64] FLAG: --qos-reserved="" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121216 2563 flags.go:64] FLAG: --read-only-port="10255" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121219 2563 flags.go:64] FLAG: --register-node="true" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121222 2563 flags.go:64] FLAG: --register-with-taints="" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121226 2563 flags.go:64] FLAG: --registry-burst="10" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121229 2563 flags.go:64] FLAG: --registry-qps="5" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121232 2563 flags.go:64] FLAG: --reserved-cpus="" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121235 2563 flags.go:64] FLAG: --reserved-memory="" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121239 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121242 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121245 2563 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121248 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121251 2563 flags.go:64] FLAG: --runonce="false" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121255 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121259 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121262 2563 flags.go:64] FLAG: --seccomp-default="false" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121264 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 08:48:07.130508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121267 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121270 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121273 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121276 2563 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121279 2563 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121282 2563 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121285 2563 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121288 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121291 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121294 2563 flags.go:64] FLAG: --system-cgroups="" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121297 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121303 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121306 2563 flags.go:64] FLAG: --tls-cert-file="" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121309 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121314 2563 flags.go:64] FLAG: --tls-min-version="" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121317 2563 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121320 2563 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121322 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121325 2563 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121328 2563 flags.go:64] FLAG: --v="2" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121332 2563 flags.go:64] FLAG: --version="false" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121337 2563 flags.go:64] FLAG: --vmodule="" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121341 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121344 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121480 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:48:07.131404 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121485 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121489 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121492 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121496 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121499 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121501 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121504 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121509 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121513 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121516 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121519 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121522 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121525 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121527 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121530 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121534 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121538 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121541 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:48:07.132435 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121544 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121547 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121549 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121552 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121555 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121557 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121560 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121562 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121565 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121568 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121570 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121573 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121576 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121578 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121581 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121583 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121586 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121590 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121592 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121595 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:48:07.132956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121597 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121601 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121604 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121606 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121609 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121612 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121614 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121617 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121619 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121622 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121625 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121627 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121630 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121632 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121635 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121638 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121640 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121643 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121646 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121648 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:48:07.133652 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121651 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121653 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121656 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121659 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121662 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121666 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121669 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121672 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121675 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121679 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121682 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121685 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121688 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121692 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121695 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121698 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121701 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121704 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121706 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:48:07.134141 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121709 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121711 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121714 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121717 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121720 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121722 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121725 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121728 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121730 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121733 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121735 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121738 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121740 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121743 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121745 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121748 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121751 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121753 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121757 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121759 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:48:07.134818 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.121762 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:48:07.135411 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.121768 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 08:48:07.135411 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.129188 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 08:48:07.135411 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.129210 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 08:48:07.135411 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.129236 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 08:48:07.135411 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.129249 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 08:48:07.135411 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129333 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:48:07.135411 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129343 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:48:07.135411 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129349 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:48:07.135411 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129354 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:48:07.135411 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129376 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:48:07.135411 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129381 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:48:07.135411 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129385 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:48:07.135411 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129390 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:48:07.135411 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129394 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:48:07.135411 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129398 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129402 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129406 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129411 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129414 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129418 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129422 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129427 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129431 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129436 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129440 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129445 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129449 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129454 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129458 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129462 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129466 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129470 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129475 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129481 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:48:07.135816 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129486 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129492 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129496 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129501 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129505 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129509 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129514 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129518 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129522 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129526 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129533 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129539 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129544 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129549 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129553 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129558 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129562 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129567 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129571 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:48:07.136416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129575 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129580 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129584 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129588 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129592 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129596 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129601 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129606 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129612 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129619 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129624 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129629 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129633 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129639 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129644 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129648 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129653 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129657 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129662 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129666 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:48:07.137196 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129670 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129675 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129679 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129683 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129687 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129691 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129696 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129700 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129705 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129709 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129713 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129718 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129722 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129726 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129730 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129735 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129739 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129743 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129747 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129752 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:48:07.137948 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129757 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:48:07.138599 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129761 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:48:07.138599 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129765 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:48:07.138599 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129769 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:48:07.138599 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129774 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:48:07.138599 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129779 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:48:07.138599 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129785 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:48:07.138599 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129792 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:48:07.138599 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129797 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:48:07.138599 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129801 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:48:07.138599 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129806 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:48:07.138599 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.129814 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 08:48:07.138599 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.129999 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 08:48:07.138599 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130006 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 08:48:07.138599 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130011 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130018 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130024 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130029 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130035 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130039 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130044 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130048 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130053 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130057 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130062 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130067 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130071 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130075 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130080 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130084 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130088 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130092 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130096 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 08:48:07.139153 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130100 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130104 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130110 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130114 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130118 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130123 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130129 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130135 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130140 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130144 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130149 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130153 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130158 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130162 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130167 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130173 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130179 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130183 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130187 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 08:48:07.139883 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130192 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130196 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130201 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130205 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130209 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130213 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130217 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130222 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130226 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130230 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130234 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130238 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130243 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130247 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130251 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130255 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130260 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130264 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130268 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130272 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 08:48:07.140345 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130277 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130282 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130286 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130290 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130294 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130298 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130302 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130306 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130310 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130315 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130319 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130323 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130328 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130332 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130337 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130341 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130345 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130349 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130353 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130357 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 08:48:07.140863 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130379 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130383 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130387 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130392 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130396 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130400 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130404 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130408 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130413 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130417 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130422 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130426 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130430 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130434 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130439 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130443 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130447 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130451 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 08:48:07.141335 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:07.130456 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 08:48:07.141787 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.130464 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 08:48:07.141787 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.130619 2563 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 08:48:07.141787 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.133098 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 08:48:07.141787 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.134263 2563 server.go:1019] "Starting client certificate rotation" Jul 29 08:48:07.141787 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.134369 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 08:48:07.141787 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.134436 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 08:48:07.141787 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.136875 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 08:48:07.141787 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.140829 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 08:48:07.147597 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.147579 2563 log.go:25] "Validated CRI v1 runtime API" Jul 29 08:48:07.150117 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.150101 2563 log.go:25] "Validated CRI v1 image API" Jul 29 08:48:07.150949 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.150931 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 08:48:07.153258 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.153239 2563 fs.go:135] Filesystem UUIDs: map[644e2c5d-e9b2-4862-9ecf-6bafb7d9d89b:/dev/nvme0n1p4 6a447300-b5f4-45c0-a9ce-63efb688cdce:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 29 08:48:07.153319 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.153258 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 08:48:07.158134 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.158021 2563 manager.go:216] Machine: {Timestamp:2026-07-29 08:48:07.156888072 +0000 UTC m=+0.280988862 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100557 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2851f2133b7a8e41a4f430f8d97796 SystemUUID:ec2851f2-133b-7a8e-41a4-f430f8d97796 BootID:8f6925cd-3f0e-4692-92be-a918163c616b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:39:fc:b3:40:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:39:fc:b3:40:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:22:14:84:f6:20 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 08:48:07.158134 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.158125 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 08:48:07.158248 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.158206 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 08:48:07.160483 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.160459 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 08:48:07.160630 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.160485 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-66.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 08:48:07.161097 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.161087 2563 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 08:48:07.161127 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.161098 2563 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 08:48:07.161127 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.161106 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 08:48:07.161540 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.161529 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 08:48:07.162100 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.162091 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 08:48:07.164774 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.164761 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 29 08:48:07.164904 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.164894 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 08:48:07.165018 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.165009 2563 kubelet.go:486] "Attempting to sync node with API server" Jul 29 08:48:07.165051 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.165022 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 08:48:07.165051 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.165037 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 08:48:07.165051 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.165046 2563 kubelet.go:398] "Adding apiserver pod source" Jul 29 08:48:07.165171 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.165054 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 08:48:07.165461 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.165450 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 08:48:07.165532 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.165476 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 08:48:07.167398 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.167381 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 08:48:07.167754 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.167733 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 08:48:07.169064 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.169048 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 08:48:07.169128 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.169079 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 08:48:07.170505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.170494 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 08:48:07.170560 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.170510 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 08:48:07.170560 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.170516 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 08:48:07.170560 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.170521 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 08:48:07.170560 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.170530 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 08:48:07.170560 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.170536 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 08:48:07.170560 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.170546 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 08:48:07.170560 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.170551 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 08:48:07.170560 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.170557 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 08:48:07.170754 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.170572 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 08:48:07.170754 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.170587 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 08:48:07.170754 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.170593 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 08:48:07.171412 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.171402 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 08:48:07.171446 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.171414 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 08:48:07.174436 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.174423 2563 server.go:1268] "Started kubelet" Jul 29 08:48:07.174602 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.174551 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 08:48:07.174602 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.174590 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 08:48:07.174706 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.174624 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 08:48:07.174706 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.174635 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 08:48:07.174706 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.174691 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 08:48:07.176030 ip-10-0-141-66 systemd[1]: Started Kubernetes Kubelet. Jul 29 08:48:07.177335 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.176090 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-66.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 08:48:07.177335 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.176980 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 08:48:07.178453 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.178423 2563 server.go:310] "Adding debug handlers to kubelet server" Jul 29 08:48:07.181943 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.181921 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 08:48:07.182959 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.182946 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 08:48:07.183024 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.182955 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 08:48:07.183067 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.183036 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Jul 29 08:48:07.183748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.183562 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 08:48:07.183748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.183581 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 08:48:07.183748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.183562 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 08:48:07.183748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.183738 2563 reconstruct.go:97] "Volume reconstruction finished" Jul 29 08:48:07.183748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.183745 2563 reconciler.go:29] "Reconciler: start to sync state" Jul 29 08:48:07.184033 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.183961 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 08:48:07.184033 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.183975 2563 factory.go:55] Registering systemd factory Jul 29 08:48:07.184033 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.183985 2563 factory.go:223] Registration of the systemd container factory successfully Jul 29 08:48:07.184178 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.184164 2563 factory.go:153] Registering CRI-O factory Jul 29 08:48:07.184210 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.184181 2563 factory.go:223] Registration of the crio container factory successfully Jul 29 08:48:07.184210 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.184205 2563 factory.go:103] Registering Raw factory Jul 29 08:48:07.184273 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.184219 2563 manager.go:1196] Started watching for new ooms in manager Jul 29 08:48:07.184575 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.184562 2563 manager.go:349] Starting recovery of all containers Jul 29 08:48:07.184676 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.183682 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-66.ec2.internal.18c6b66e4e3f2c67 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-66.ec2.internal,UID:ip-10-0-141-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-66.ec2.internal,},FirstTimestamp:2026-07-29 08:48:07.174401127 +0000 UTC m=+0.298501917,LastTimestamp:2026-07-29 08:48:07.174401127 +0000 UTC m=+0.298501917,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-66.ec2.internal,}" Jul 29 08:48:07.189385 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.189343 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-65c88" Jul 29 08:48:07.190301 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.190264 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 08:48:07.194462 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.194439 2563 manager.go:354] Recovery completed Jul 29 08:48:07.197086 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.197065 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-65c88" Jul 29 08:48:07.198095 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.198069 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 08:48:07.198181 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.198126 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 08:48:07.198929 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.198918 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:48:07.200827 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.200813 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:48:07.200890 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.200841 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:48:07.200890 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.200854 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:48:07.201409 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.201391 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 08:48:07.201409 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.201401 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:48:07.201409 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.201407 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 08:48:07.201586 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.201424 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 29 08:48:07.202145 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.202128 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:48:07.202220 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.202163 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:48:07.202220 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.202172 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:48:07.203751 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.203739 2563 policy_none.go:49] "None policy: Start" Jul 29 08:48:07.203789 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.203755 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 08:48:07.203789 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.203768 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 08:48:07.205695 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.205681 2563 policy_none.go:47] "Start" Jul 29 08:48:07.252095 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.251947 2563 manager.go:337] "Starting Device Plugin manager" Jul 29 08:48:07.252194 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.252118 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 08:48:07.252194 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.252128 2563 server.go:87] "Starting device plugin registration server" Jul 29 08:48:07.252374 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.252350 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 08:48:07.252406 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.252378 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 08:48:07.252476 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.252457 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 08:48:07.252565 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.252546 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 08:48:07.252565 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.252560 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 08:48:07.253095 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.253072 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 08:48:07.253165 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.253124 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-66.ec2.internal\" not found" Jul 29 08:48:07.352623 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.352589 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:48:07.354743 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.353679 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:48:07.354743 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.353708 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:48:07.354743 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.353718 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:48:07.354743 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.353746 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.355656 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.355633 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 08:48:07.355746 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.355672 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 08:48:07.355746 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.355692 2563 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 08:48:07.355746 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.355723 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 08:48:07.357452 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.357434 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:48:07.360814 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.360750 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.360814 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.360773 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-66.ec2.internal\": node \"ip-10-0-141-66.ec2.internal\" not found" Jul 29 08:48:07.433204 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.433172 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Jul 29 08:48:07.456294 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.456245 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal"] Jul 29 08:48:07.456408 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.456354 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:48:07.457845 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.457828 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:48:07.457954 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.457861 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:48:07.457954 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.457876 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:48:07.460180 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.460165 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:48:07.460295 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.460281 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.460342 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.460319 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:48:07.461153 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.461138 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:48:07.461232 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.461168 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:48:07.461232 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.461182 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:48:07.461707 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.461692 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:48:07.461771 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.461724 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:48:07.461771 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.461739 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:48:07.463812 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.463798 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.463881 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.463822 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 08:48:07.464510 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.464494 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientMemory" Jul 29 08:48:07.464578 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.464525 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 08:48:07.464578 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.464542 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeHasSufficientPID" Jul 29 08:48:07.484564 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.484544 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/537cd980128174ada17eaa7ddacad7c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"537cd980128174ada17eaa7ddacad7c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.484677 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.484573 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/537cd980128174ada17eaa7ddacad7c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"537cd980128174ada17eaa7ddacad7c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.484677 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.484597 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f6d3bed9801bf6ab9cc93c6f1440b4f0-config\") pod \"kube-apiserver-proxy-ip-10-0-141-66.ec2.internal\" (UID: \"f6d3bed9801bf6ab9cc93c6f1440b4f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.494108 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.494067 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-66.ec2.internal\" not found" node="ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.498572 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.498558 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-66.ec2.internal\" not found" node="ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.534050 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.534029 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Jul 29 08:48:07.585262 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.585228 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/537cd980128174ada17eaa7ddacad7c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"537cd980128174ada17eaa7ddacad7c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.585262 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.585250 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/537cd980128174ada17eaa7ddacad7c4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"537cd980128174ada17eaa7ddacad7c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.585451 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.585277 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/537cd980128174ada17eaa7ddacad7c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"537cd980128174ada17eaa7ddacad7c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.585451 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.585300 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f6d3bed9801bf6ab9cc93c6f1440b4f0-config\") pod \"kube-apiserver-proxy-ip-10-0-141-66.ec2.internal\" (UID: \"f6d3bed9801bf6ab9cc93c6f1440b4f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.585451 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.585323 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/537cd980128174ada17eaa7ddacad7c4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal\" (UID: \"537cd980128174ada17eaa7ddacad7c4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.585451 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.585340 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f6d3bed9801bf6ab9cc93c6f1440b4f0-config\") pod \"kube-apiserver-proxy-ip-10-0-141-66.ec2.internal\" (UID: \"f6d3bed9801bf6ab9cc93c6f1440b4f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.635702 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.635608 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Jul 29 08:48:07.739137 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.739097 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Jul 29 08:48:07.796846 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.796818 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.800745 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:07.800718 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" Jul 29 08:48:07.839834 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.839798 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Jul 29 08:48:07.940073 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:07.939985 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Jul 29 08:48:08.040497 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:08.040454 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Jul 29 08:48:08.136619 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:08.136584 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 08:48:08.137140 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:08.136750 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 08:48:08.141145 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:08.141122 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Jul 29 08:48:08.202160 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:08.202064 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 08:43:07 +0000 UTC" deadline="2028-01-28 12:13:29.560188692 +0000 UTC" Jul 29 08:48:08.202160 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:08.202099 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13155h25m21.358095057s" Jul 29 08:48:08.241529 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:08.241497 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-66.ec2.internal\" not found" Jul 29 08:48:08.329060 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:08.329022 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:48:08.383562 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:08.383522 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" Jul 29 08:48:08.388535 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:08.388515 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:48:08.393368 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:08.393350 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 08:48:08.395354 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:08.395333 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" Jul 29 08:48:08.396919 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:08.396892 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d3bed9801bf6ab9cc93c6f1440b4f0.slice/crio-490e9aa97a421265493318e5d746b9bd8b842794fcfdfb849ec884710f1919ca WatchSource:0}: Error finding container 490e9aa97a421265493318e5d746b9bd8b842794fcfdfb849ec884710f1919ca: Status 404 returned error can't find the container with id 490e9aa97a421265493318e5d746b9bd8b842794fcfdfb849ec884710f1919ca Jul 29 08:48:08.397304 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:08.397286 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537cd980128174ada17eaa7ddacad7c4.slice/crio-4ca86761464c1f57c5a92cd05c75b4fd5913ab0ddd68558881d4a684f3735437 WatchSource:0}: Error finding container 4ca86761464c1f57c5a92cd05c75b4fd5913ab0ddd68558881d4a684f3735437: Status 404 returned error can't find the container with id 4ca86761464c1f57c5a92cd05c75b4fd5913ab0ddd68558881d4a684f3735437 Jul 29 08:48:08.402740 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:08.402723 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:48:08.409521 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:08.409504 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 08:48:08.571126 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:08.569701 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:48:09.090694 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.090664 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 08:48:09.166455 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.166424 2563 apiserver.go:52] "Watching apiserver" Jul 29 08:48:09.177084 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.177060 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 08:48:09.178850 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.178828 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg","openshift-cluster-node-tuning-operator/tuned-xd5rj","openshift-dns/node-resolver-hmb59","openshift-image-registry/node-ca-cpx2c","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal","openshift-multus/network-metrics-daemon-9dgqn","openshift-network-operator/iptables-alerter-zdvqz","kube-system/global-pull-secret-syncer-z6kw7","kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal","openshift-multus/multus-65p4s","openshift-multus/multus-additional-cni-plugins-hvlsg","openshift-network-diagnostics/network-check-target-rwtwp","openshift-ovn-kubernetes/ovnkube-node-s7zc5","kube-system/konnectivity-agent-vmwkf"] Jul 29 08:48:09.182834 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.182817 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zdvqz" Jul 29 08:48:09.184311 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.184287 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 08:48:09.185056 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.185038 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.185945 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.185929 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 08:48:09.186066 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.186039 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 08:48:09.186142 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.186076 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 08:48:09.186199 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.186184 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g25pz\"" Jul 29 08:48:09.187268 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.187249 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hmb59" Jul 29 08:48:09.189459 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.189437 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cpx2c" Jul 29 08:48:09.189542 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.189476 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 08:48:09.189819 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.189800 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 08:48:09.190015 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.189997 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 08:48:09.190077 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.189997 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 08:48:09.190490 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.190425 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 08:48:09.190490 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.190436 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sdz2x\"" Jul 29 08:48:09.191123 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.191103 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 08:48:09.191209 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.191131 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qjsd5\"" Jul 29 08:48:09.191209 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.191111 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 08:48:09.191708 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.191692 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vmwkf" Jul 29 08:48:09.193572 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.193555 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 08:48:09.193675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.193587 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 08:48:09.194029 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.194011 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:09.194104 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:09.194085 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:09.194196 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.194178 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0f11c605-0ff0-4365-b858-56741a112444-iptables-alerter-script\") pod \"iptables-alerter-zdvqz\" (UID: \"0f11c605-0ff0-4365-b858-56741a112444\") " pod="openshift-network-operator/iptables-alerter-zdvqz" Jul 29 08:48:09.194259 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.194211 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f11c605-0ff0-4365-b858-56741a112444-host-slash\") pod \"iptables-alerter-zdvqz\" (UID: \"0f11c605-0ff0-4365-b858-56741a112444\") " pod="openshift-network-operator/iptables-alerter-zdvqz" Jul 29 08:48:09.194259 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.194246 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/510492e8-2257-458d-aff3-b69dd243e9d5-cnibin\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.194680 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.194653 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/510492e8-2257-458d-aff3-b69dd243e9d5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.194748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.194698 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/510492e8-2257-458d-aff3-b69dd243e9d5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.194809 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.194725 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/510492e8-2257-458d-aff3-b69dd243e9d5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.194809 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.194779 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2f2s\" (UniqueName: \"kubernetes.io/projected/510492e8-2257-458d-aff3-b69dd243e9d5-kube-api-access-s2f2s\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.194903 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.194815 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpsft\" (UniqueName: \"kubernetes.io/projected/0f11c605-0ff0-4365-b858-56741a112444-kube-api-access-dpsft\") pod \"iptables-alerter-zdvqz\" (UID: \"0f11c605-0ff0-4365-b858-56741a112444\") " pod="openshift-network-operator/iptables-alerter-zdvqz" Jul 29 08:48:09.194952 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.194913 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/510492e8-2257-458d-aff3-b69dd243e9d5-system-cni-dir\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.195007 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.194956 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/510492e8-2257-458d-aff3-b69dd243e9d5-os-release\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.195007 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.194989 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/510492e8-2257-458d-aff3-b69dd243e9d5-cni-binary-copy\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.195111 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.195049 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 08:48:09.195111 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.195100 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n9xdh\"" Jul 29 08:48:09.195700 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.195684 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lhdzr\"" Jul 29 08:48:09.195786 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.195719 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 08:48:09.196262 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.196213 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 08:48:09.196478 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.196457 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:09.196563 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:09.196538 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:09.200688 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.200666 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.203977 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.203952 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.205401 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.205231 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 08:48:09.205401 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.205235 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 08:48:09.205401 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.205280 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 08:48:09.205401 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.205289 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vx29d\"" Jul 29 08:48:09.205401 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.205302 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 08:48:09.205401 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.205324 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 08:48:09.205401 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.205232 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 08:48:09.205715 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.205413 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 08:48:09.206235 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.206214 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.207129 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.207108 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lkkjz\"" Jul 29 08:48:09.207386 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.207366 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 08:48:09.207386 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.207376 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 08:48:09.208869 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.208851 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.210545 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.210357 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-npfdz\"" Jul 29 08:48:09.210545 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.210413 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 08:48:09.211333 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.211312 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:09.211426 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:09.211387 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:09.212062 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.212035 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sbht5\"" Jul 29 08:48:09.212062 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.212053 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 08:48:09.212210 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.212043 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 08:48:09.212210 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.212106 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 08:48:09.227617 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.227594 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nm5p9" Jul 29 08:48:09.237569 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.237546 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nm5p9" Jul 29 08:48:09.284863 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.284836 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 08:48:09.295986 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.295960 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-system-cni-dir\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.296116 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296000 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-var-lib-kubelet\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.296116 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296064 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.296116 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296103 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-etc-kubernetes\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.296292 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296129 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9014122b-1078-4e35-b53c-234e93bbd054-dbus\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:09.296292 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296153 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9165852-3048-4abd-8dde-d2b364c5a28f-ovnkube-script-lib\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.296292 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296179 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-tuned\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.296292 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296211 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-var-lib-cni-bin\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.296292 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296231 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-multus-conf-dir\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.296292 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296261 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-socket-dir\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.296292 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296287 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vwn\" (UniqueName: \"kubernetes.io/projected/ee96c05e-aba5-4248-91e6-9cba3f709801-kube-api-access-v4vwn\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.296675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296313 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pclsb\" (UniqueName: \"kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb\") pod \"network-check-target-rwtwp\" (UID: \"3e7aa462-6508-4fa0-ab6d-43a5670db158\") " pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:09.296675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296337 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-var-lib-openvswitch\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.296675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296356 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vz28\" (UniqueName: \"kubernetes.io/projected/86239d8c-0cc4-4be4-b21a-e31d5139934c-kube-api-access-8vz28\") pod \"node-resolver-hmb59\" (UID: \"86239d8c-0cc4-4be4-b21a-e31d5139934c\") " pod="openshift-dns/node-resolver-hmb59" Jul 29 08:48:09.296675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296370 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-run-k8s-cni-cncf-io\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.296675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296386 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.296675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296414 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-systemd\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.296675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296433 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f11c605-0ff0-4365-b858-56741a112444-host-slash\") pod \"iptables-alerter-zdvqz\" (UID: \"0f11c605-0ff0-4365-b858-56741a112444\") " pod="openshift-network-operator/iptables-alerter-zdvqz" Jul 29 08:48:09.296675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296456 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs\") pod \"network-metrics-daemon-9dgqn\" (UID: \"d5fac53f-30ee-496f-9131-c819ae8e4963\") " pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:09.296675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296518 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-device-dir\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.296675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296588 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1422c6a2-0287-4e57-8379-b18d6fcef17e-host\") pod \"node-ca-cpx2c\" (UID: \"1422c6a2-0287-4e57-8379-b18d6fcef17e\") " pod="openshift-image-registry/node-ca-cpx2c" Jul 29 08:48:09.296675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296625 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-run-multus-certs\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.296675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296595 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f11c605-0ff0-4365-b858-56741a112444-host-slash\") pod \"iptables-alerter-zdvqz\" (UID: \"0f11c605-0ff0-4365-b858-56741a112444\") " pod="openshift-network-operator/iptables-alerter-zdvqz" Jul 29 08:48:09.296675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296663 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-log-socket\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296691 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-sys\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296707 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-var-lib-kubelet\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296749 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f22q\" (UniqueName: \"kubernetes.io/projected/8213caf5-2d71-44ec-b4cc-0e0db2157f42-kube-api-access-2f22q\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296786 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c7837bb5-8d62-4a06-9e62-15ff27aea228-multus-daemon-config\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296818 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-etc-selinux\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296855 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/510492e8-2257-458d-aff3-b69dd243e9d5-os-release\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296882 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-etc-openvswitch\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296915 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-cnibin\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.296980 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-run-netns\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297009 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/86239d8c-0cc4-4be4-b21a-e31d5139934c-hosts-file\") pod \"node-resolver-hmb59\" (UID: \"86239d8c-0cc4-4be4-b21a-e31d5139934c\") " pod="openshift-dns/node-resolver-hmb59" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297025 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7837bb5-8d62-4a06-9e62-15ff27aea228-cni-binary-copy\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297056 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-multus-socket-dir-parent\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297080 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-var-lib-cni-multus\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297080 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/510492e8-2257-458d-aff3-b69dd243e9d5-os-release\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297095 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-registration-dir\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297113 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/510492e8-2257-458d-aff3-b69dd243e9d5-system-cni-dir\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.297241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297142 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-kubelet\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297175 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/510492e8-2257-458d-aff3-b69dd243e9d5-system-cni-dir\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297213 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8213caf5-2d71-44ec-b4cc-0e0db2157f42-tmp\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297240 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9014122b-1078-4e35-b53c-234e93bbd054-kubelet-config\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297273 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-systemd-units\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297297 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9165852-3048-4abd-8dde-d2b364c5a28f-ovnkube-config\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297322 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scxhc\" (UniqueName: \"kubernetes.io/projected/c7837bb5-8d62-4a06-9e62-15ff27aea228-kube-api-access-scxhc\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297346 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/97483690-18fa-4576-801e-4de241e61858-agent-certs\") pod \"konnectivity-agent-vmwkf\" (UID: \"97483690-18fa-4576-801e-4de241e61858\") " pod="kube-system/konnectivity-agent-vmwkf" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297403 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-run-openvswitch\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297440 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-run-ovn\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297464 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-modprobe-d\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297496 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-sys-fs\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297526 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/510492e8-2257-458d-aff3-b69dd243e9d5-cnibin\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297556 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/510492e8-2257-458d-aff3-b69dd243e9d5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297611 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/510492e8-2257-458d-aff3-b69dd243e9d5-cnibin\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297644 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/510492e8-2257-458d-aff3-b69dd243e9d5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.298072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297671 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/510492e8-2257-458d-aff3-b69dd243e9d5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297717 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2f2s\" (UniqueName: \"kubernetes.io/projected/510492e8-2257-458d-aff3-b69dd243e9d5-kube-api-access-s2f2s\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297768 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-slash\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297801 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-run-systemd\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297870 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/510492e8-2257-458d-aff3-b69dd243e9d5-cni-binary-copy\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297898 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-cni-bin\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297928 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86239d8c-0cc4-4be4-b21a-e31d5139934c-tmp-dir\") pod \"node-resolver-hmb59\" (UID: \"86239d8c-0cc4-4be4-b21a-e31d5139934c\") " pod="openshift-dns/node-resolver-hmb59" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.297997 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1422c6a2-0287-4e57-8379-b18d6fcef17e-serviceca\") pod \"node-ca-cpx2c\" (UID: \"1422c6a2-0287-4e57-8379-b18d6fcef17e\") " pod="openshift-image-registry/node-ca-cpx2c" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298025 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlkdd\" (UniqueName: \"kubernetes.io/projected/1422c6a2-0287-4e57-8379-b18d6fcef17e-kube-api-access-vlkdd\") pod \"node-ca-cpx2c\" (UID: \"1422c6a2-0287-4e57-8379-b18d6fcef17e\") " pod="openshift-image-registry/node-ca-cpx2c" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298051 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-sysconfig\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298076 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-host\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298137 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-multus-cni-dir\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298172 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vflb\" (UniqueName: \"kubernetes.io/projected/d5fac53f-30ee-496f-9131-c819ae8e4963-kube-api-access-4vflb\") pod \"network-metrics-daemon-9dgqn\" (UID: \"d5fac53f-30ee-496f-9131-c819ae8e4963\") " pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298201 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298297 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-run-netns\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298297 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/510492e8-2257-458d-aff3-b69dd243e9d5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.298924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298334 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jf9h\" (UniqueName: \"kubernetes.io/projected/b9165852-3048-4abd-8dde-d2b364c5a28f-kube-api-access-6jf9h\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298365 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-sysctl-d\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298392 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-os-release\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298418 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-hostroot\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298548 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/510492e8-2257-458d-aff3-b69dd243e9d5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298606 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-sysctl-conf\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298638 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-run\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298674 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpsft\" (UniqueName: \"kubernetes.io/projected/0f11c605-0ff0-4365-b858-56741a112444-kube-api-access-dpsft\") pod \"iptables-alerter-zdvqz\" (UID: \"0f11c605-0ff0-4365-b858-56741a112444\") " pod="openshift-network-operator/iptables-alerter-zdvqz" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298712 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298742 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9165852-3048-4abd-8dde-d2b364c5a28f-env-overrides\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298772 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9165852-3048-4abd-8dde-d2b364c5a28f-ovn-node-metrics-cert\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298803 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-kubernetes\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298834 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0f11c605-0ff0-4365-b858-56741a112444-iptables-alerter-script\") pod \"iptables-alerter-zdvqz\" (UID: \"0f11c605-0ff0-4365-b858-56741a112444\") " pod="openshift-network-operator/iptables-alerter-zdvqz" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298858 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/510492e8-2257-458d-aff3-b69dd243e9d5-cni-binary-copy\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.298867 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/97483690-18fa-4576-801e-4de241e61858-konnectivity-ca\") pod \"konnectivity-agent-vmwkf\" (UID: \"97483690-18fa-4576-801e-4de241e61858\") " pod="kube-system/konnectivity-agent-vmwkf" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.299222 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-node-log\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.299770 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.299257 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-cni-netd\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.300587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.299285 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-lib-modules\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.300587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.299405 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/510492e8-2257-458d-aff3-b69dd243e9d5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.300587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.299865 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0f11c605-0ff0-4365-b858-56741a112444-iptables-alerter-script\") pod \"iptables-alerter-zdvqz\" (UID: \"0f11c605-0ff0-4365-b858-56741a112444\") " pod="openshift-network-operator/iptables-alerter-zdvqz" Jul 29 08:48:09.310131 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.310101 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 08:48:09.314479 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.314457 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpsft\" (UniqueName: \"kubernetes.io/projected/0f11c605-0ff0-4365-b858-56741a112444-kube-api-access-dpsft\") pod \"iptables-alerter-zdvqz\" (UID: \"0f11c605-0ff0-4365-b858-56741a112444\") " pod="openshift-network-operator/iptables-alerter-zdvqz" Jul 29 08:48:09.314579 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.314568 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2f2s\" (UniqueName: \"kubernetes.io/projected/510492e8-2257-458d-aff3-b69dd243e9d5-kube-api-access-s2f2s\") pod \"multus-additional-cni-plugins-hvlsg\" (UID: \"510492e8-2257-458d-aff3-b69dd243e9d5\") " pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.362393 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.362254 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" event={"ID":"537cd980128174ada17eaa7ddacad7c4","Type":"ContainerStarted","Data":"4ca86761464c1f57c5a92cd05c75b4fd5913ab0ddd68558881d4a684f3735437"} Jul 29 08:48:09.363747 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.363696 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" event={"ID":"f6d3bed9801bf6ab9cc93c6f1440b4f0","Type":"ContainerStarted","Data":"490e9aa97a421265493318e5d746b9bd8b842794fcfdfb849ec884710f1919ca"} Jul 29 08:48:09.399848 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.399817 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-multus-cni-dir\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.400001 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.399861 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vflb\" (UniqueName: \"kubernetes.io/projected/d5fac53f-30ee-496f-9131-c819ae8e4963-kube-api-access-4vflb\") pod \"network-metrics-daemon-9dgqn\" (UID: \"d5fac53f-30ee-496f-9131-c819ae8e4963\") " pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:09.400001 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.399889 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:09.400001 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.399916 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-run-netns\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.400001 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.399942 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf9h\" (UniqueName: \"kubernetes.io/projected/b9165852-3048-4abd-8dde-d2b364c5a28f-kube-api-access-6jf9h\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.400001 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.399969 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-sysctl-d\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.400272 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.399996 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-os-release\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.400272 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400034 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-hostroot\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.400272 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-sysctl-conf\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.400272 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:09.400073 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:48:09.400272 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:09.400171 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret podName:9014122b-1078-4e35-b53c-234e93bbd054 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:09.900145738 +0000 UTC m=+3.024246521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret") pod "global-pull-secret-syncer-z6kw7" (UID: "9014122b-1078-4e35-b53c-234e93bbd054") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:48:09.400272 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400082 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-run\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.400272 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400204 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-run\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.400272 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400212 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-multus-cni-dir\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.400272 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400230 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.400272 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400257 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-run-netns\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.400272 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400264 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9165852-3048-4abd-8dde-d2b364c5a28f-env-overrides\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9165852-3048-4abd-8dde-d2b364c5a28f-ovn-node-metrics-cert\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400323 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-kubernetes\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400374 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/97483690-18fa-4576-801e-4de241e61858-konnectivity-ca\") pod \"konnectivity-agent-vmwkf\" (UID: \"97483690-18fa-4576-801e-4de241e61858\") " pod="kube-system/konnectivity-agent-vmwkf" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400392 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-sysctl-conf\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400401 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-node-log\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400428 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-cni-netd\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400455 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-hostroot\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400469 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-sysctl-d\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400499 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400466 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-lib-modules\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400501 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-os-release\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400537 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-system-cni-dir\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400552 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-node-log\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400566 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-var-lib-kubelet\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400586 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-lib-modules\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400631 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-etc-kubernetes\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.400748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400656 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9014122b-1078-4e35-b53c-234e93bbd054-dbus\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400665 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-cni-netd\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400684 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9165852-3048-4abd-8dde-d2b364c5a28f-ovnkube-script-lib\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400707 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-run-ovn-kubernetes\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400710 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-tuned\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400742 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-var-lib-cni-bin\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400771 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-multus-conf-dir\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400800 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-socket-dir\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400827 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vwn\" (UniqueName: \"kubernetes.io/projected/ee96c05e-aba5-4248-91e6-9cba3f709801-kube-api-access-v4vwn\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400858 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclsb\" (UniqueName: \"kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb\") pod \"network-check-target-rwtwp\" (UID: \"3e7aa462-6508-4fa0-ab6d-43a5670db158\") " pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400891 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-var-lib-openvswitch\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400913 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-var-lib-cni-bin\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400964 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-multus-conf-dir\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400987 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b9165852-3048-4abd-8dde-d2b364c5a28f-env-overrides\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401054 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/97483690-18fa-4576-801e-4de241e61858-konnectivity-ca\") pod \"konnectivity-agent-vmwkf\" (UID: \"97483690-18fa-4576-801e-4de241e61858\") " pod="kube-system/konnectivity-agent-vmwkf" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401078 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-socket-dir\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401102 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9014122b-1078-4e35-b53c-234e93bbd054-dbus\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:09.401587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401112 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-var-lib-kubelet\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401125 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-var-lib-openvswitch\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401153 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-etc-kubernetes\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.400919 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vz28\" (UniqueName: \"kubernetes.io/projected/86239d8c-0cc4-4be4-b21a-e31d5139934c-kube-api-access-8vz28\") pod \"node-resolver-hmb59\" (UID: \"86239d8c-0cc4-4be4-b21a-e31d5139934c\") " pod="openshift-dns/node-resolver-hmb59" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401168 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-system-cni-dir\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401259 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-run-k8s-cni-cncf-io\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401306 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401339 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-systemd\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401358 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-run-k8s-cni-cncf-io\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401391 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs\") pod \"network-metrics-daemon-9dgqn\" (UID: \"d5fac53f-30ee-496f-9131-c819ae8e4963\") " pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401409 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401400 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-kubernetes\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401424 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-device-dir\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401454 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1422c6a2-0287-4e57-8379-b18d6fcef17e-host\") pod \"node-ca-cpx2c\" (UID: \"1422c6a2-0287-4e57-8379-b18d6fcef17e\") " pod="openshift-image-registry/node-ca-cpx2c" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401473 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-systemd\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401484 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-run-multus-certs\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401497 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-device-dir\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:09.401509 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:48:09.402505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401531 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1422c6a2-0287-4e57-8379-b18d6fcef17e-host\") pod \"node-ca-cpx2c\" (UID: \"1422c6a2-0287-4e57-8379-b18d6fcef17e\") " pod="openshift-image-registry/node-ca-cpx2c" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401512 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-log-socket\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401549 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-run-multus-certs\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401553 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-log-socket\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:09.401568 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs podName:d5fac53f-30ee-496f-9131-c819ae8e4963 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:09.901546438 +0000 UTC m=+3.025647231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs") pod "network-metrics-daemon-9dgqn" (UID: "d5fac53f-30ee-496f-9131-c819ae8e4963") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401611 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-sys\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401646 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-var-lib-kubelet\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401648 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b9165852-3048-4abd-8dde-d2b364c5a28f-ovnkube-script-lib\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401673 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f22q\" (UniqueName: \"kubernetes.io/projected/8213caf5-2d71-44ec-b4cc-0e0db2157f42-kube-api-access-2f22q\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401699 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c7837bb5-8d62-4a06-9e62-15ff27aea228-multus-daemon-config\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401695 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-sys\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401712 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-var-lib-kubelet\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401725 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-etc-selinux\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401755 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-etc-openvswitch\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401785 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-cnibin\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401810 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-run-netns\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401844 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/86239d8c-0cc4-4be4-b21a-e31d5139934c-hosts-file\") pod \"node-resolver-hmb59\" (UID: \"86239d8c-0cc4-4be4-b21a-e31d5139934c\") " pod="openshift-dns/node-resolver-hmb59" Jul 29 08:48:09.403485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401871 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7837bb5-8d62-4a06-9e62-15ff27aea228-cni-binary-copy\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401902 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-multus-socket-dir-parent\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401923 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-cnibin\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401930 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-var-lib-cni-multus\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401968 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-registration-dir\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401998 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-kubelet\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402013 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-run-netns\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402024 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8213caf5-2d71-44ec-b4cc-0e0db2157f42-tmp\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402052 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9014122b-1078-4e35-b53c-234e93bbd054-kubelet-config\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402073 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-multus-socket-dir-parent\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402079 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-systemd-units\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402106 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9165852-3048-4abd-8dde-d2b364c5a28f-ovnkube-config\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402120 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-kubelet\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402132 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scxhc\" (UniqueName: \"kubernetes.io/projected/c7837bb5-8d62-4a06-9e62-15ff27aea228-kube-api-access-scxhc\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402159 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/97483690-18fa-4576-801e-4de241e61858-agent-certs\") pod \"konnectivity-agent-vmwkf\" (UID: \"97483690-18fa-4576-801e-4de241e61858\") " pod="kube-system/konnectivity-agent-vmwkf" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402165 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-etc-openvswitch\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402183 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-run-openvswitch\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402207 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-run-ovn\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.404228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402222 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-registration-dir\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402232 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-modprobe-d\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.401969 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c7837bb5-8d62-4a06-9e62-15ff27aea228-host-var-lib-cni-multus\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402261 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c7837bb5-8d62-4a06-9e62-15ff27aea228-multus-daemon-config\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402024 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-etc-selinux\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402339 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-run-ovn\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402386 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-run-openvswitch\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402505 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-modprobe-d\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402573 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9014122b-1078-4e35-b53c-234e93bbd054-kubelet-config\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402628 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/86239d8c-0cc4-4be4-b21a-e31d5139934c-hosts-file\") pod \"node-resolver-hmb59\" (UID: \"86239d8c-0cc4-4be4-b21a-e31d5139934c\") " pod="openshift-dns/node-resolver-hmb59" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402637 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-systemd-units\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402756 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-sys-fs\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402779 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b9165852-3048-4abd-8dde-d2b364c5a28f-ovnkube-config\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402797 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-slash\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402824 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-run-systemd\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402857 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ee96c05e-aba5-4248-91e6-9cba3f709801-sys-fs\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402853 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-cni-bin\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402908 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-slash\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.405075 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402911 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86239d8c-0cc4-4be4-b21a-e31d5139934c-tmp-dir\") pod \"node-resolver-hmb59\" (UID: \"86239d8c-0cc4-4be4-b21a-e31d5139934c\") " pod="openshift-dns/node-resolver-hmb59" Jul 29 08:48:09.405837 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.402978 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1422c6a2-0287-4e57-8379-b18d6fcef17e-serviceca\") pod \"node-ca-cpx2c\" (UID: \"1422c6a2-0287-4e57-8379-b18d6fcef17e\") " pod="openshift-image-registry/node-ca-cpx2c" Jul 29 08:48:09.405837 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.403012 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlkdd\" (UniqueName: \"kubernetes.io/projected/1422c6a2-0287-4e57-8379-b18d6fcef17e-kube-api-access-vlkdd\") pod \"node-ca-cpx2c\" (UID: \"1422c6a2-0287-4e57-8379-b18d6fcef17e\") " pod="openshift-image-registry/node-ca-cpx2c" Jul 29 08:48:09.405837 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.403040 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-sysconfig\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.405837 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.403259 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86239d8c-0cc4-4be4-b21a-e31d5139934c-tmp-dir\") pod \"node-resolver-hmb59\" (UID: \"86239d8c-0cc4-4be4-b21a-e31d5139934c\") " pod="openshift-dns/node-resolver-hmb59" Jul 29 08:48:09.405837 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.403289 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-host\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.405837 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.403294 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-run-systemd\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.405837 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.403362 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b9165852-3048-4abd-8dde-d2b364c5a28f-host-cni-bin\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.405837 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.403395 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-sysconfig\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.405837 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.403471 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8213caf5-2d71-44ec-b4cc-0e0db2157f42-host\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.405837 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.403567 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1422c6a2-0287-4e57-8379-b18d6fcef17e-serviceca\") pod \"node-ca-cpx2c\" (UID: \"1422c6a2-0287-4e57-8379-b18d6fcef17e\") " pod="openshift-image-registry/node-ca-cpx2c" Jul 29 08:48:09.405837 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.403601 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c7837bb5-8d62-4a06-9e62-15ff27aea228-cni-binary-copy\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.405837 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.403612 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b9165852-3048-4abd-8dde-d2b364c5a28f-ovn-node-metrics-cert\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.405837 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.404115 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8213caf5-2d71-44ec-b4cc-0e0db2157f42-etc-tuned\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.405837 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.405459 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8213caf5-2d71-44ec-b4cc-0e0db2157f42-tmp\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.407288 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.407265 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/97483690-18fa-4576-801e-4de241e61858-agent-certs\") pod \"konnectivity-agent-vmwkf\" (UID: \"97483690-18fa-4576-801e-4de241e61858\") " pod="kube-system/konnectivity-agent-vmwkf" Jul 29 08:48:09.412341 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.412283 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vflb\" (UniqueName: \"kubernetes.io/projected/d5fac53f-30ee-496f-9131-c819ae8e4963-kube-api-access-4vflb\") pod \"network-metrics-daemon-9dgqn\" (UID: \"d5fac53f-30ee-496f-9131-c819ae8e4963\") " pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:09.412683 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:09.412656 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:48:09.412827 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:09.412686 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:48:09.412827 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:09.412701 2563 projected.go:196] Error preparing data for projected volume kube-api-access-pclsb for pod openshift-network-diagnostics/network-check-target-rwtwp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:48:09.412827 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:09.412772 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb podName:3e7aa462-6508-4fa0-ab6d-43a5670db158 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:09.912755614 +0000 UTC m=+3.036856412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pclsb" (UniqueName: "kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb") pod "network-check-target-rwtwp" (UID: "3e7aa462-6508-4fa0-ab6d-43a5670db158") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:48:09.414734 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.414705 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vwn\" (UniqueName: \"kubernetes.io/projected/ee96c05e-aba5-4248-91e6-9cba3f709801-kube-api-access-v4vwn\") pod \"aws-ebs-csi-driver-node-6pdzg\" (UID: \"ee96c05e-aba5-4248-91e6-9cba3f709801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.415363 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.415342 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f22q\" (UniqueName: \"kubernetes.io/projected/8213caf5-2d71-44ec-b4cc-0e0db2157f42-kube-api-access-2f22q\") pod \"tuned-xd5rj\" (UID: \"8213caf5-2d71-44ec-b4cc-0e0db2157f42\") " pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.416387 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.416062 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jf9h\" (UniqueName: \"kubernetes.io/projected/b9165852-3048-4abd-8dde-d2b364c5a28f-kube-api-access-6jf9h\") pod \"ovnkube-node-s7zc5\" (UID: \"b9165852-3048-4abd-8dde-d2b364c5a28f\") " pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.416629 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.416604 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlkdd\" (UniqueName: \"kubernetes.io/projected/1422c6a2-0287-4e57-8379-b18d6fcef17e-kube-api-access-vlkdd\") pod \"node-ca-cpx2c\" (UID: \"1422c6a2-0287-4e57-8379-b18d6fcef17e\") " pod="openshift-image-registry/node-ca-cpx2c" Jul 29 08:48:09.417149 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.417125 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vz28\" (UniqueName: \"kubernetes.io/projected/86239d8c-0cc4-4be4-b21a-e31d5139934c-kube-api-access-8vz28\") pod \"node-resolver-hmb59\" (UID: \"86239d8c-0cc4-4be4-b21a-e31d5139934c\") " pod="openshift-dns/node-resolver-hmb59" Jul 29 08:48:09.430484 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.430458 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scxhc\" (UniqueName: \"kubernetes.io/projected/c7837bb5-8d62-4a06-9e62-15ff27aea228-kube-api-access-scxhc\") pod \"multus-65p4s\" (UID: \"c7837bb5-8d62-4a06-9e62-15ff27aea228\") " pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.494111 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.494019 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zdvqz" Jul 29 08:48:09.503317 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.503283 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hvlsg" Jul 29 08:48:09.507658 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:09.507624 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f11c605_0ff0_4365_b858_56741a112444.slice/crio-12b2a268d4e1104f820766d50e7faba495007ed81a5e69ee25c8f4f919640d19 WatchSource:0}: Error finding container 12b2a268d4e1104f820766d50e7faba495007ed81a5e69ee25c8f4f919640d19: Status 404 returned error can't find the container with id 12b2a268d4e1104f820766d50e7faba495007ed81a5e69ee25c8f4f919640d19 Jul 29 08:48:09.511573 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.511482 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hmb59" Jul 29 08:48:09.515387 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:09.515359 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510492e8_2257_458d_aff3_b69dd243e9d5.slice/crio-db44e7a80d87c43a865b9976d6f17eec32f3b2665e83e225a700394e6736ffd9 WatchSource:0}: Error finding container db44e7a80d87c43a865b9976d6f17eec32f3b2665e83e225a700394e6736ffd9: Status 404 returned error can't find the container with id db44e7a80d87c43a865b9976d6f17eec32f3b2665e83e225a700394e6736ffd9 Jul 29 08:48:09.517625 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.517554 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cpx2c" Jul 29 08:48:09.522661 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:09.522633 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86239d8c_0cc4_4be4_b21a_e31d5139934c.slice/crio-5195e884f3fdf179e1889d51ceebb0be4a7827d76a9788322f880151e23ebcd0 WatchSource:0}: Error finding container 5195e884f3fdf179e1889d51ceebb0be4a7827d76a9788322f880151e23ebcd0: Status 404 returned error can't find the container with id 5195e884f3fdf179e1889d51ceebb0be4a7827d76a9788322f880151e23ebcd0 Jul 29 08:48:09.524694 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.524681 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vmwkf" Jul 29 08:48:09.528599 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:09.528570 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1422c6a2_0287_4e57_8379_b18d6fcef17e.slice/crio-61205f289363d1d1bcf41ad831c48a1c95a1939c7d239bc60447b3f8f2ab5fff WatchSource:0}: Error finding container 61205f289363d1d1bcf41ad831c48a1c95a1939c7d239bc60447b3f8f2ab5fff: Status 404 returned error can't find the container with id 61205f289363d1d1bcf41ad831c48a1c95a1939c7d239bc60447b3f8f2ab5fff Jul 29 08:48:09.535501 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:09.535474 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97483690_18fa_4576_801e_4de241e61858.slice/crio-672cbb7fd167c1ad44c87e3b8d7c47b9b47912d139ab08a68e09bf2408c0d0f4 WatchSource:0}: Error finding container 672cbb7fd167c1ad44c87e3b8d7c47b9b47912d139ab08a68e09bf2408c0d0f4: Status 404 returned error can't find the container with id 672cbb7fd167c1ad44c87e3b8d7c47b9b47912d139ab08a68e09bf2408c0d0f4 Jul 29 08:48:09.538997 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.538972 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:09.546766 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.546739 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" Jul 29 08:48:09.553404 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:09.553358 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9165852_3048_4abd_8dde_d2b364c5a28f.slice/crio-3d54231d0d52305fd8531b9f34c9372c9ebd86ffcabf433458ac1b3af4a5e95e WatchSource:0}: Error finding container 3d54231d0d52305fd8531b9f34c9372c9ebd86ffcabf433458ac1b3af4a5e95e: Status 404 returned error can't find the container with id 3d54231d0d52305fd8531b9f34c9372c9ebd86ffcabf433458ac1b3af4a5e95e Jul 29 08:48:09.554334 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.554308 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-65p4s" Jul 29 08:48:09.559390 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.559366 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" Jul 29 08:48:09.559711 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:09.559686 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8213caf5_2d71_44ec_b4cc_0e0db2157f42.slice/crio-415c1483f5130d21286026eb10a9c65969d992af318620fa60cec46f5666e87a WatchSource:0}: Error finding container 415c1483f5130d21286026eb10a9c65969d992af318620fa60cec46f5666e87a: Status 404 returned error can't find the container with id 415c1483f5130d21286026eb10a9c65969d992af318620fa60cec46f5666e87a Jul 29 08:48:09.567416 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:09.567388 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7837bb5_8d62_4a06_9e62_15ff27aea228.slice/crio-86d21b7753a8fe6471bcea6b3d2192cbc78eea1d4ade640d7a84c584cc7c9a9c WatchSource:0}: Error finding container 86d21b7753a8fe6471bcea6b3d2192cbc78eea1d4ade640d7a84c584cc7c9a9c: Status 404 returned error can't find the container with id 86d21b7753a8fe6471bcea6b3d2192cbc78eea1d4ade640d7a84c584cc7c9a9c Jul 29 08:48:09.570903 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:09.570879 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee96c05e_aba5_4248_91e6_9cba3f709801.slice/crio-f34fa23352aa36272912c1ca3ea8a14030471bf1e64eae935870f04d5b9a6162 WatchSource:0}: Error finding container f34fa23352aa36272912c1ca3ea8a14030471bf1e64eae935870f04d5b9a6162: Status 404 returned error can't find the container with id f34fa23352aa36272912c1ca3ea8a14030471bf1e64eae935870f04d5b9a6162 Jul 29 08:48:09.906305 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.906265 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs\") pod \"network-metrics-daemon-9dgqn\" (UID: \"d5fac53f-30ee-496f-9131-c819ae8e4963\") " pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:09.906539 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:09.906345 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:09.906539 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:09.906493 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:48:09.906641 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:09.906547 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret podName:9014122b-1078-4e35-b53c-234e93bbd054 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:10.90653343 +0000 UTC m=+4.030634213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret") pod "global-pull-secret-syncer-z6kw7" (UID: "9014122b-1078-4e35-b53c-234e93bbd054") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:48:09.906835 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:09.906795 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:48:09.906985 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:09.906881 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs podName:d5fac53f-30ee-496f-9131-c819ae8e4963 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:10.906857334 +0000 UTC m=+4.030958118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs") pod "network-metrics-daemon-9dgqn" (UID: "d5fac53f-30ee-496f-9131-c819ae8e4963") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:48:10.007249 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:10.007215 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclsb\" (UniqueName: \"kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb\") pod \"network-check-target-rwtwp\" (UID: \"3e7aa462-6508-4fa0-ab6d-43a5670db158\") " pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:10.007473 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:10.007435 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:48:10.007473 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:10.007461 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:48:10.007473 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:10.007473 2563 projected.go:196] Error preparing data for projected volume kube-api-access-pclsb for pod openshift-network-diagnostics/network-check-target-rwtwp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:48:10.007674 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:10.007552 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb podName:3e7aa462-6508-4fa0-ab6d-43a5670db158 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:11.007534108 +0000 UTC m=+4.131634896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pclsb" (UniqueName: "kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb") pod "network-check-target-rwtwp" (UID: "3e7aa462-6508-4fa0-ab6d-43a5670db158") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:48:10.241220 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:10.241102 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 08:43:09 +0000 UTC" deadline="2028-02-23 22:44:35.388681903 +0000 UTC" Jul 29 08:48:10.241220 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:10.241160 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13789h56m25.147527033s" Jul 29 08:48:10.366713 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:10.366672 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" event={"ID":"ee96c05e-aba5-4248-91e6-9cba3f709801","Type":"ContainerStarted","Data":"f34fa23352aa36272912c1ca3ea8a14030471bf1e64eae935870f04d5b9a6162"} Jul 29 08:48:10.368135 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:10.368100 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" event={"ID":"b9165852-3048-4abd-8dde-d2b364c5a28f","Type":"ContainerStarted","Data":"3d54231d0d52305fd8531b9f34c9372c9ebd86ffcabf433458ac1b3af4a5e95e"} Jul 29 08:48:10.369428 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:10.369380 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vmwkf" event={"ID":"97483690-18fa-4576-801e-4de241e61858","Type":"ContainerStarted","Data":"672cbb7fd167c1ad44c87e3b8d7c47b9b47912d139ab08a68e09bf2408c0d0f4"} Jul 29 08:48:10.370561 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:10.370537 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cpx2c" event={"ID":"1422c6a2-0287-4e57-8379-b18d6fcef17e","Type":"ContainerStarted","Data":"61205f289363d1d1bcf41ad831c48a1c95a1939c7d239bc60447b3f8f2ab5fff"} Jul 29 08:48:10.371752 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:10.371722 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-65p4s" event={"ID":"c7837bb5-8d62-4a06-9e62-15ff27aea228","Type":"ContainerStarted","Data":"86d21b7753a8fe6471bcea6b3d2192cbc78eea1d4ade640d7a84c584cc7c9a9c"} Jul 29 08:48:10.372916 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:10.372895 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" event={"ID":"8213caf5-2d71-44ec-b4cc-0e0db2157f42","Type":"ContainerStarted","Data":"415c1483f5130d21286026eb10a9c65969d992af318620fa60cec46f5666e87a"} Jul 29 08:48:10.374004 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:10.373981 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hmb59" event={"ID":"86239d8c-0cc4-4be4-b21a-e31d5139934c","Type":"ContainerStarted","Data":"5195e884f3fdf179e1889d51ceebb0be4a7827d76a9788322f880151e23ebcd0"} Jul 29 08:48:10.374972 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:10.374952 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvlsg" event={"ID":"510492e8-2257-458d-aff3-b69dd243e9d5","Type":"ContainerStarted","Data":"db44e7a80d87c43a865b9976d6f17eec32f3b2665e83e225a700394e6736ffd9"} Jul 29 08:48:10.375942 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:10.375923 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zdvqz" event={"ID":"0f11c605-0ff0-4365-b858-56741a112444","Type":"ContainerStarted","Data":"12b2a268d4e1104f820766d50e7faba495007ed81a5e69ee25c8f4f919640d19"} Jul 29 08:48:10.916927 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:10.916874 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs\") pod \"network-metrics-daemon-9dgqn\" (UID: \"d5fac53f-30ee-496f-9131-c819ae8e4963\") " pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:10.917122 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:10.916969 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:10.917122 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:10.917091 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:48:10.917250 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:10.917161 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret podName:9014122b-1078-4e35-b53c-234e93bbd054 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:12.917139505 +0000 UTC m=+6.041240287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret") pod "global-pull-secret-syncer-z6kw7" (UID: "9014122b-1078-4e35-b53c-234e93bbd054") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:48:10.917250 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:10.917091 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:48:10.917250 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:10.917245 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs podName:d5fac53f-30ee-496f-9131-c819ae8e4963 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:12.91723072 +0000 UTC m=+6.041331504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs") pod "network-metrics-daemon-9dgqn" (UID: "d5fac53f-30ee-496f-9131-c819ae8e4963") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:48:11.017455 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:11.017388 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclsb\" (UniqueName: \"kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb\") pod \"network-check-target-rwtwp\" (UID: \"3e7aa462-6508-4fa0-ab6d-43a5670db158\") " pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:11.017668 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:11.017646 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:48:11.017755 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:11.017681 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:48:11.017755 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:11.017698 2563 projected.go:196] Error preparing data for projected volume kube-api-access-pclsb for pod openshift-network-diagnostics/network-check-target-rwtwp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:48:11.017875 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:11.017796 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb podName:3e7aa462-6508-4fa0-ab6d-43a5670db158 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:13.017764913 +0000 UTC m=+6.141865719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pclsb" (UniqueName: "kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb") pod "network-check-target-rwtwp" (UID: "3e7aa462-6508-4fa0-ab6d-43a5670db158") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:48:11.242113 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:11.242018 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 08:43:09 +0000 UTC" deadline="2028-05-14 18:14:48.262908698 +0000 UTC" Jul 29 08:48:11.242113 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:11.242060 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15729h26m37.020854241s" Jul 29 08:48:11.358932 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:11.358897 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:11.359219 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:11.359187 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:11.359817 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:11.359778 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:11.359935 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:11.359911 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:11.360146 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:11.360128 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:11.360265 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:11.360242 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:12.407774 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:12.407375 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" event={"ID":"f6d3bed9801bf6ab9cc93c6f1440b4f0","Type":"ContainerStarted","Data":"5e05dfcba59de0c8fe1afed7ca55421db7cd98d31e673b3fcef2ee9c0dfdfe1b"} Jul 29 08:48:12.420070 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:12.419856 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" event={"ID":"537cd980128174ada17eaa7ddacad7c4","Type":"ContainerStarted","Data":"085c55006b0e65f297c3ccae85edea5e211d5f290a0d66a416b479511157097a"} Jul 29 08:48:12.446523 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:12.446431 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-66.ec2.internal" podStartSLOduration=3.44640447 podStartE2EDuration="3.44640447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:12.425403599 +0000 UTC m=+5.549504405" watchObservedRunningTime="2026-07-29 08:48:12.44640447 +0000 UTC m=+5.570505274" Jul 29 08:48:12.935271 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:12.935231 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:12.935559 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:12.935318 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs\") pod \"network-metrics-daemon-9dgqn\" (UID: \"d5fac53f-30ee-496f-9131-c819ae8e4963\") " pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:12.935559 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:12.935480 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:48:12.935559 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:12.935544 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs podName:d5fac53f-30ee-496f-9131-c819ae8e4963 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:16.935522893 +0000 UTC m=+10.059623673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs") pod "network-metrics-daemon-9dgqn" (UID: "d5fac53f-30ee-496f-9131-c819ae8e4963") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:48:12.936284 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:12.935969 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:48:12.936284 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:12.936025 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret podName:9014122b-1078-4e35-b53c-234e93bbd054 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:16.936009086 +0000 UTC m=+10.060109865 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret") pod "global-pull-secret-syncer-z6kw7" (UID: "9014122b-1078-4e35-b53c-234e93bbd054") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:48:13.037557 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:13.036733 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclsb\" (UniqueName: \"kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb\") pod \"network-check-target-rwtwp\" (UID: \"3e7aa462-6508-4fa0-ab6d-43a5670db158\") " pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:13.037557 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:13.036937 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:48:13.037557 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:13.036961 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:48:13.037557 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:13.036975 2563 projected.go:196] Error preparing data for projected volume kube-api-access-pclsb for pod openshift-network-diagnostics/network-check-target-rwtwp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:48:13.037557 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:13.037062 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb podName:3e7aa462-6508-4fa0-ab6d-43a5670db158 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:17.037022145 +0000 UTC m=+10.161122942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pclsb" (UniqueName: "kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb") pod "network-check-target-rwtwp" (UID: "3e7aa462-6508-4fa0-ab6d-43a5670db158") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:48:13.359533 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:13.358692 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:13.359533 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:13.358823 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:13.359533 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:13.358906 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:13.359533 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:13.358992 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:13.359533 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:13.359042 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:13.359533 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:13.359107 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:15.358517 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:15.357544 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:15.358517 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:15.357682 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:15.358517 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:15.358088 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:15.358517 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:15.358213 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:15.359103 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:15.358634 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:15.359103 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:15.358747 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:15.432495 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:15.432318 2563 generic.go:356] "Generic (PLEG): container finished" podID="537cd980128174ada17eaa7ddacad7c4" containerID="085c55006b0e65f297c3ccae85edea5e211d5f290a0d66a416b479511157097a" exitCode=0 Jul 29 08:48:15.432495 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:15.432384 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" event={"ID":"537cd980128174ada17eaa7ddacad7c4","Type":"ContainerDied","Data":"085c55006b0e65f297c3ccae85edea5e211d5f290a0d66a416b479511157097a"} Jul 29 08:48:16.974732 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:16.974688 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:16.975192 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:16.974764 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs\") pod \"network-metrics-daemon-9dgqn\" (UID: \"d5fac53f-30ee-496f-9131-c819ae8e4963\") " pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:16.975192 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:16.974860 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:48:16.975192 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:16.974918 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs podName:d5fac53f-30ee-496f-9131-c819ae8e4963 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:24.97490302 +0000 UTC m=+18.099003803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs") pod "network-metrics-daemon-9dgqn" (UID: "d5fac53f-30ee-496f-9131-c819ae8e4963") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:48:16.975192 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:16.974860 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:48:16.975192 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:16.974995 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret podName:9014122b-1078-4e35-b53c-234e93bbd054 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:24.974980452 +0000 UTC m=+18.099081237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret") pod "global-pull-secret-syncer-z6kw7" (UID: "9014122b-1078-4e35-b53c-234e93bbd054") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:48:17.075482 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:17.075435 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclsb\" (UniqueName: \"kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb\") pod \"network-check-target-rwtwp\" (UID: \"3e7aa462-6508-4fa0-ab6d-43a5670db158\") " pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:17.075691 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:17.075633 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:48:17.075691 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:17.075655 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:48:17.075691 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:17.075669 2563 projected.go:196] Error preparing data for projected volume kube-api-access-pclsb for pod openshift-network-diagnostics/network-check-target-rwtwp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:48:17.075882 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:17.075742 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb podName:3e7aa462-6508-4fa0-ab6d-43a5670db158 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:25.075723181 +0000 UTC m=+18.199823963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pclsb" (UniqueName: "kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb") pod "network-check-target-rwtwp" (UID: "3e7aa462-6508-4fa0-ab6d-43a5670db158") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:48:17.357960 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:17.357414 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:17.357960 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:17.357549 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:17.357960 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:17.357556 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:17.357960 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:17.357728 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:17.357960 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:17.357743 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:17.357960 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:17.357827 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:19.356819 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:19.356777 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:19.357398 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:19.356925 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:19.357398 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:19.356978 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:19.357398 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:19.357074 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:19.357561 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:19.357403 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:19.357561 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:19.357508 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:21.356918 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:21.356886 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:21.357312 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:21.356930 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:21.357312 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:21.356994 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:21.357312 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:21.357094 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:21.357514 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:21.357336 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:21.357514 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:21.357484 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:23.355916 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:23.355886 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:23.360729 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:23.355886 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:23.360729 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:23.356018 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:23.360729 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:23.355886 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:23.360729 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:23.356097 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:23.360729 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:23.356164 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:25.038655 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:25.038621 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:25.039054 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:25.038678 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs\") pod \"network-metrics-daemon-9dgqn\" (UID: \"d5fac53f-30ee-496f-9131-c819ae8e4963\") " pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:25.039054 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:25.038779 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:48:25.039054 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:25.038835 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs podName:d5fac53f-30ee-496f-9131-c819ae8e4963 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:41.038821744 +0000 UTC m=+34.162922520 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs") pod "network-metrics-daemon-9dgqn" (UID: "d5fac53f-30ee-496f-9131-c819ae8e4963") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:48:25.039054 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:25.038780 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:48:25.039054 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:25.038914 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret podName:9014122b-1078-4e35-b53c-234e93bbd054 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:41.038900247 +0000 UTC m=+34.163001042 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret") pod "global-pull-secret-syncer-z6kw7" (UID: "9014122b-1078-4e35-b53c-234e93bbd054") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:48:25.139322 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:25.139284 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclsb\" (UniqueName: \"kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb\") pod \"network-check-target-rwtwp\" (UID: \"3e7aa462-6508-4fa0-ab6d-43a5670db158\") " pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:25.139490 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:25.139471 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:48:25.139529 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:25.139495 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:48:25.139529 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:25.139506 2563 projected.go:196] Error preparing data for projected volume kube-api-access-pclsb for pod openshift-network-diagnostics/network-check-target-rwtwp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:48:25.139588 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:25.139556 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb podName:3e7aa462-6508-4fa0-ab6d-43a5670db158 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:41.139543093 +0000 UTC m=+34.263643870 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pclsb" (UniqueName: "kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb") pod "network-check-target-rwtwp" (UID: "3e7aa462-6508-4fa0-ab6d-43a5670db158") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:48:25.356333 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:25.356225 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:25.356333 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:25.356226 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:25.356570 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:25.356356 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:25.356570 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:25.356233 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:25.356570 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:25.356458 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:25.356570 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:25.356533 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:27.357195 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:27.357154 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:27.357653 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:27.357208 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:27.357653 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:27.357273 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:27.357653 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:27.357329 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:27.357653 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:27.357376 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:27.357653 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:27.357475 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:29.356476 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.356261 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:29.357308 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.356262 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:29.357308 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:29.356583 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:29.357308 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.356262 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:29.357308 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:29.356662 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:29.357308 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:29.356756 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:29.460927 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.460888 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vmwkf" event={"ID":"97483690-18fa-4576-801e-4de241e61858","Type":"ContainerStarted","Data":"bbd01243505f74f7b9a10d5ccc7cca96056f8e42795eeefa4127a4821e932f11"} Jul 29 08:48:29.462506 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.462481 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cpx2c" event={"ID":"1422c6a2-0287-4e57-8379-b18d6fcef17e","Type":"ContainerStarted","Data":"0c3a52881baa5db1e41dc3a32ce2a34ec4f21eb686ad96a6403b50dc568c4945"} Jul 29 08:48:29.464486 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.464460 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" event={"ID":"537cd980128174ada17eaa7ddacad7c4","Type":"ContainerStarted","Data":"bbad95074eb6ebc2fc2a63565a33478703ecb3185eafbd953b5428c7a6070fca"} Jul 29 08:48:29.466059 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.466026 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-65p4s" event={"ID":"c7837bb5-8d62-4a06-9e62-15ff27aea228","Type":"ContainerStarted","Data":"c033eba6571ca60d5d0d0bda0fe55345a3372c3f2cf553794fff949d77312245"} Jul 29 08:48:29.467662 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.467637 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" event={"ID":"8213caf5-2d71-44ec-b4cc-0e0db2157f42","Type":"ContainerStarted","Data":"881db98d559e3f125f0fc0d6b5ed15cb0f87d71e84bceda7efd63ff2ebeaec32"} Jul 29 08:48:29.469089 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.469060 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hmb59" event={"ID":"86239d8c-0cc4-4be4-b21a-e31d5139934c","Type":"ContainerStarted","Data":"6d1a66cab9d44e872631376b9612d9b2a3f9c4119930a992bc3f840edbb3b674"} Jul 29 08:48:29.470602 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.470577 2563 generic.go:356] "Generic (PLEG): container finished" podID="510492e8-2257-458d-aff3-b69dd243e9d5" containerID="f77e13964bda07a9c2fe3e157cb1388bf9ab4fa2021685efe968ab6c840988f2" exitCode=0 Jul 29 08:48:29.470675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.470649 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvlsg" event={"ID":"510492e8-2257-458d-aff3-b69dd243e9d5","Type":"ContainerDied","Data":"f77e13964bda07a9c2fe3e157cb1388bf9ab4fa2021685efe968ab6c840988f2"} Jul 29 08:48:29.472654 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.472636 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" event={"ID":"ee96c05e-aba5-4248-91e6-9cba3f709801","Type":"ContainerStarted","Data":"367a96c7d6b02611ba2ae85391722d720b69328cead44896c1194f4983b14880"} Jul 29 08:48:29.474729 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.474711 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" event={"ID":"b9165852-3048-4abd-8dde-d2b364c5a28f","Type":"ContainerStarted","Data":"2c1922e872f30e8d881ba4fc418e6f31e9c9e2b415ff43fb1ffadda0bcc0365a"} Jul 29 08:48:29.474806 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.474766 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" event={"ID":"b9165852-3048-4abd-8dde-d2b364c5a28f","Type":"ContainerStarted","Data":"f403a97ef3e73ac08f716c5c70034c8a707b29e54570a9583575ccea413131be"} Jul 29 08:48:29.474806 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.474798 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" event={"ID":"b9165852-3048-4abd-8dde-d2b364c5a28f","Type":"ContainerStarted","Data":"cef3b0d3439505c7e403e8c10e96b99b582baa14a8ac2295c525771db7969821"} Jul 29 08:48:29.476046 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.476012 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vmwkf" podStartSLOduration=1.995738318 podStartE2EDuration="21.475999803s" totalImagesPullingTime="19.480261485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:29.475519139 +0000 UTC m=+22.599619938" watchObservedRunningTime="2026-07-29 08:48:29.475999803 +0000 UTC m=+22.600100602" Jul 29 08:48:29.492940 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.492904 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-66.ec2.internal" podStartSLOduration=20.492891433 podStartE2EDuration="20.492891433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:29.492594699 +0000 UTC m=+22.616695499" watchObservedRunningTime="2026-07-29 08:48:29.492891433 +0000 UTC m=+22.616992231" Jul 29 08:48:29.515197 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.515175 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vmwkf" Jul 29 08:48:29.515681 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.515665 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vmwkf" Jul 29 08:48:29.527432 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.527397 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hmb59" podStartSLOduration=2.015023312 podStartE2EDuration="21.527381023s" totalImagesPullingTime="19.512357711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:29.527028487 +0000 UTC m=+22.651129285" watchObservedRunningTime="2026-07-29 08:48:29.527381023 +0000 UTC m=+22.651481823" Jul 29 08:48:29.544003 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.543962 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cpx2c" podStartSLOduration=2.035508394 podStartE2EDuration="21.543947011s" totalImagesPullingTime="19.508438617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:29.543336593 +0000 UTC m=+22.667437609" watchObservedRunningTime="2026-07-29 08:48:29.543947011 +0000 UTC m=+22.668047815" Jul 29 08:48:29.561660 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.561611 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xd5rj" podStartSLOduration=2.081830665 podStartE2EDuration="21.561596889s" totalImagesPullingTime="19.479766224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:29.560867256 +0000 UTC m=+22.684968054" watchObservedRunningTime="2026-07-29 08:48:29.561596889 +0000 UTC m=+22.685697687" Jul 29 08:48:29.579802 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:29.579757 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-65p4s" podStartSLOduration=2.056657425 podStartE2EDuration="21.579741335s" totalImagesPullingTime="19.52308391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:29.579214861 +0000 UTC m=+22.703315660" watchObservedRunningTime="2026-07-29 08:48:29.579741335 +0000 UTC m=+22.703842134" Jul 29 08:48:30.480505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:30.480460 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" event={"ID":"b9165852-3048-4abd-8dde-d2b364c5a28f","Type":"ContainerStarted","Data":"0b9ab67823e1b66371226704cbd5b888df0509347b183195ca98cb593bc0c9f6"} Jul 29 08:48:30.481045 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:30.480578 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" event={"ID":"b9165852-3048-4abd-8dde-d2b364c5a28f","Type":"ContainerStarted","Data":"1707acc22dd2ac542fed2b51dfdcf97993c129c62d26028b22a6e6591369b34e"} Jul 29 08:48:30.481045 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:30.480611 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" event={"ID":"b9165852-3048-4abd-8dde-d2b364c5a28f","Type":"ContainerStarted","Data":"d6068f53040cafe41c39f4f303f76e06c954dcae90fbad68bdab8f245472ae99"} Jul 29 08:48:30.482445 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:30.482413 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zdvqz" event={"ID":"0f11c605-0ff0-4365-b858-56741a112444","Type":"ContainerStarted","Data":"5d1a49bbf7ab64c0202d0d0dd12384688b0bb0c03aecb7e2ce8e8fa7827fcff8"} Jul 29 08:48:30.517544 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:30.517494 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zdvqz" podStartSLOduration=2.992540812 podStartE2EDuration="22.517476838s" totalImagesPullingTime="19.524936026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:30.516590256 +0000 UTC m=+23.640691057" watchObservedRunningTime="2026-07-29 08:48:30.517476838 +0000 UTC m=+23.641577667" Jul 29 08:48:30.683765 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:30.683729 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 08:48:31.268724 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:31.268598 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T08:48:30.683752183Z","UUID":"de0577aa-428a-417b-b0cd-8b3df38cb0fd","Handler":null,"Name":"","Endpoint":""} Jul 29 08:48:31.270417 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:31.270390 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 08:48:31.270549 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:31.270434 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 08:48:31.356467 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:31.356438 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:31.356642 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:31.356438 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:31.356642 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:31.356553 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:31.356642 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:31.356581 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:31.356809 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:31.356686 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:31.356809 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:31.356784 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:31.487721 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:31.487679 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" event={"ID":"ee96c05e-aba5-4248-91e6-9cba3f709801","Type":"ContainerStarted","Data":"24ba3878ae5a525c63bf0278da79e7541ee25ea815607a95dde6066409060613"} Jul 29 08:48:31.488189 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:31.487854 2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 29 08:48:32.460525 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:32.460479 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vmwkf" Jul 29 08:48:32.461064 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:32.461042 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vmwkf" Jul 29 08:48:32.492000 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:32.491965 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" event={"ID":"ee96c05e-aba5-4248-91e6-9cba3f709801","Type":"ContainerStarted","Data":"c5e1db40ebec434e3cf21add163fe62ed69cbf14c2d1915cfbc2e801839114e8"} Jul 29 08:48:32.495241 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:32.495214 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" event={"ID":"b9165852-3048-4abd-8dde-d2b364c5a28f","Type":"ContainerStarted","Data":"774a003dbf305222615c864ba179cba5aed6de025268a3e2e25f9db3f4994b3c"} Jul 29 08:48:33.355969 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:33.355935 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:33.356152 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:33.355935 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:33.356152 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:33.356075 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:33.356152 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:33.356094 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:33.356300 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:33.356221 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:33.356382 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:33.356305 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:34.503784 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:34.503580 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" event={"ID":"b9165852-3048-4abd-8dde-d2b364c5a28f","Type":"ContainerStarted","Data":"6a282c4de00596736faa32fdb9fe88e30a173dac1101de66a7520555782d7cb5"} Jul 29 08:48:34.504670 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:34.503837 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:34.505572 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:34.505543 2563 generic.go:356] "Generic (PLEG): container finished" podID="510492e8-2257-458d-aff3-b69dd243e9d5" containerID="9e8ed8988e0f57b45ac382960ac5cc800f7ac4b99307ad2cd423d1e5bae4b2e8" exitCode=0 Jul 29 08:48:34.505717 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:34.505580 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvlsg" event={"ID":"510492e8-2257-458d-aff3-b69dd243e9d5","Type":"ContainerDied","Data":"9e8ed8988e0f57b45ac382960ac5cc800f7ac4b99307ad2cd423d1e5bae4b2e8"} Jul 29 08:48:34.522274 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:34.522252 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:34.534458 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:34.534415 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" podStartSLOduration=6.976221171 podStartE2EDuration="26.534400389s" totalImagesPullingTime="19.558179218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:34.534158893 +0000 UTC m=+27.658259691" watchObservedRunningTime="2026-07-29 08:48:34.534400389 +0000 UTC m=+27.658501187" Jul 29 08:48:34.534603 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:34.534585 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6pdzg" podStartSLOduration=12.307903865 podStartE2EDuration="26.534579315s" totalImagesPullingTime="14.22667545s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:48:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:32.516243607 +0000 UTC m=+25.640344405" watchObservedRunningTime="2026-07-29 08:48:34.534579315 +0000 UTC m=+27.658680114" Jul 29 08:48:35.356125 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:35.355948 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:35.356274 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:35.355947 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:35.356274 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:35.356216 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:35.356274 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:35.356260 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:35.356412 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:35.356004 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:35.356412 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:35.356333 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:35.510037 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:35.510003 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:35.510501 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:35.510101 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:35.530280 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:35.530249 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:48:36.463465 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:36.463427 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9dgqn"] Jul 29 08:48:36.463725 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:36.463554 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:36.463725 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:36.463661 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:36.464275 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:36.464245 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z6kw7"] Jul 29 08:48:36.464408 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:36.464313 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:36.464464 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:36.464411 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:36.465498 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:36.465477 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rwtwp"] Jul 29 08:48:36.465598 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:36.465547 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:36.465659 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:36.465611 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:36.512792 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:36.512717 2563 generic.go:356] "Generic (PLEG): container finished" podID="510492e8-2257-458d-aff3-b69dd243e9d5" containerID="ef3fcff4c63f00fc9120d9662f6f3b0c6615220a2796d013e5e8c659c371085b" exitCode=0 Jul 29 08:48:36.513284 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:36.512789 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvlsg" event={"ID":"510492e8-2257-458d-aff3-b69dd243e9d5","Type":"ContainerDied","Data":"ef3fcff4c63f00fc9120d9662f6f3b0c6615220a2796d013e5e8c659c371085b"} Jul 29 08:48:38.356579 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:38.356540 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:38.356945 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:38.356547 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:38.356945 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:38.356556 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:38.356945 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:38.356752 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:38.356945 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:38.356825 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:38.356945 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:38.356654 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:38.519897 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:38.519817 2563 generic.go:356] "Generic (PLEG): container finished" podID="510492e8-2257-458d-aff3-b69dd243e9d5" containerID="e9218834be1afb68650833393781ab11902cabff44b05eedaa5a522d37b0dfd0" exitCode=0 Jul 29 08:48:38.519897 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:38.519880 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvlsg" event={"ID":"510492e8-2257-458d-aff3-b69dd243e9d5","Type":"ContainerDied","Data":"e9218834be1afb68650833393781ab11902cabff44b05eedaa5a522d37b0dfd0"} Jul 29 08:48:40.356643 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:40.356411 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:40.357055 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:40.356448 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:40.357055 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:40.356744 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:40.357055 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:40.356448 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:40.357055 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:40.356821 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:40.357055 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:40.356923 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:41.050855 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:41.050821 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:41.051055 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:41.050872 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs\") pod \"network-metrics-daemon-9dgqn\" (UID: \"d5fac53f-30ee-496f-9131-c819ae8e4963\") " pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:41.051055 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:41.050997 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 08:48:41.051170 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:41.051062 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:48:41.051170 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:41.051080 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret podName:9014122b-1078-4e35-b53c-234e93bbd054 nodeName:}" failed. No retries permitted until 2026-07-29 08:49:13.051059675 +0000 UTC m=+66.175160452 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret") pod "global-pull-secret-syncer-z6kw7" (UID: "9014122b-1078-4e35-b53c-234e93bbd054") : object "kube-system"/"original-pull-secret" not registered Jul 29 08:48:41.051170 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:41.051122 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs podName:d5fac53f-30ee-496f-9131-c819ae8e4963 nodeName:}" failed. No retries permitted until 2026-07-29 08:49:13.051104135 +0000 UTC m=+66.175204919 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs") pod "network-metrics-daemon-9dgqn" (UID: "d5fac53f-30ee-496f-9131-c819ae8e4963") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 08:48:41.151583 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:41.151543 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclsb\" (UniqueName: \"kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb\") pod \"network-check-target-rwtwp\" (UID: \"3e7aa462-6508-4fa0-ab6d-43a5670db158\") " pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:41.151763 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:41.151745 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 08:48:41.151834 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:41.151772 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 08:48:41.151834 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:41.151786 2563 projected.go:196] Error preparing data for projected volume kube-api-access-pclsb for pod openshift-network-diagnostics/network-check-target-rwtwp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:48:41.151916 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:41.151861 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb podName:3e7aa462-6508-4fa0-ab6d-43a5670db158 nodeName:}" failed. No retries permitted until 2026-07-29 08:49:13.151840411 +0000 UTC m=+66.275941202 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pclsb" (UniqueName: "kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb") pod "network-check-target-rwtwp" (UID: "3e7aa462-6508-4fa0-ab6d-43a5670db158") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 08:48:42.356494 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.356459 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:42.356494 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.356482 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:42.356494 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.356493 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:42.357114 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:42.356590 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z6kw7" podUID="9014122b-1078-4e35-b53c-234e93bbd054" Jul 29 08:48:42.357114 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:42.356663 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rwtwp" podUID="3e7aa462-6508-4fa0-ab6d-43a5670db158" Jul 29 08:48:42.357114 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:42.356746 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9dgqn" podUID="d5fac53f-30ee-496f-9131-c819ae8e4963" Jul 29 08:48:42.730632 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.730609 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-66.ec2.internal" event="NodeReady" Jul 29 08:48:42.730738 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.730728 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 08:48:42.776343 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.776306 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7ff2z"] Jul 29 08:48:42.779128 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.779113 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:42.782198 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.782158 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 08:48:42.782198 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.782160 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-68zl6\"" Jul 29 08:48:42.782355 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.782164 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 08:48:42.782625 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.782610 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4pvfb"] Jul 29 08:48:42.785224 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.785205 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4pvfb" Jul 29 08:48:42.787915 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.787894 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wb7dl\"" Jul 29 08:48:42.788106 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.787930 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 08:48:42.788397 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.787997 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 08:48:42.788397 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.788098 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 08:48:42.789087 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.789068 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7ff2z"] Jul 29 08:48:42.795272 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.795254 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4pvfb"] Jul 29 08:48:42.864298 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.864217 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ffc23fe-8cc5-4d63-bc98-fba3c171561b-cert\") pod \"ingress-canary-4pvfb\" (UID: \"9ffc23fe-8cc5-4d63-bc98-fba3c171561b\") " pod="openshift-ingress-canary/ingress-canary-4pvfb" Jul 29 08:48:42.864298 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.864270 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvvzm\" (UniqueName: \"kubernetes.io/projected/bc60cc72-c068-4d53-a2ad-4141c76472a5-kube-api-access-wvvzm\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:42.864298 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.864297 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkstb\" (UniqueName: \"kubernetes.io/projected/9ffc23fe-8cc5-4d63-bc98-fba3c171561b-kube-api-access-fkstb\") pod \"ingress-canary-4pvfb\" (UID: \"9ffc23fe-8cc5-4d63-bc98-fba3c171561b\") " pod="openshift-ingress-canary/ingress-canary-4pvfb" Jul 29 08:48:42.864535 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.864339 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc60cc72-c068-4d53-a2ad-4141c76472a5-tmp-dir\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:42.864535 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.864399 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc60cc72-c068-4d53-a2ad-4141c76472a5-config-volume\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:42.864535 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.864418 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:42.965647 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.965614 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ffc23fe-8cc5-4d63-bc98-fba3c171561b-cert\") pod \"ingress-canary-4pvfb\" (UID: \"9ffc23fe-8cc5-4d63-bc98-fba3c171561b\") " pod="openshift-ingress-canary/ingress-canary-4pvfb" Jul 29 08:48:42.965767 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.965666 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvvzm\" (UniqueName: \"kubernetes.io/projected/bc60cc72-c068-4d53-a2ad-4141c76472a5-kube-api-access-wvvzm\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:42.965767 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.965696 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkstb\" (UniqueName: \"kubernetes.io/projected/9ffc23fe-8cc5-4d63-bc98-fba3c171561b-kube-api-access-fkstb\") pod \"ingress-canary-4pvfb\" (UID: \"9ffc23fe-8cc5-4d63-bc98-fba3c171561b\") " pod="openshift-ingress-canary/ingress-canary-4pvfb" Jul 29 08:48:42.965767 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.965714 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc60cc72-c068-4d53-a2ad-4141c76472a5-tmp-dir\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:42.965767 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.965738 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc60cc72-c068-4d53-a2ad-4141c76472a5-config-volume\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:42.965767 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.965755 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:42.965987 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:42.965954 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:48:42.966124 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:42.966093 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls podName:bc60cc72-c068-4d53-a2ad-4141c76472a5 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:43.466078353 +0000 UTC m=+36.590179129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls") pod "dns-default-7ff2z" (UID: "bc60cc72-c068-4d53-a2ad-4141c76472a5") : secret "dns-default-metrics-tls" not found Jul 29 08:48:42.966213 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.966125 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc60cc72-c068-4d53-a2ad-4141c76472a5-tmp-dir\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:42.966575 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.966554 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc60cc72-c068-4d53-a2ad-4141c76472a5-config-volume\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:42.969868 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.969850 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ffc23fe-8cc5-4d63-bc98-fba3c171561b-cert\") pod \"ingress-canary-4pvfb\" (UID: \"9ffc23fe-8cc5-4d63-bc98-fba3c171561b\") " pod="openshift-ingress-canary/ingress-canary-4pvfb" Jul 29 08:48:42.974987 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.974961 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkstb\" (UniqueName: \"kubernetes.io/projected/9ffc23fe-8cc5-4d63-bc98-fba3c171561b-kube-api-access-fkstb\") pod \"ingress-canary-4pvfb\" (UID: \"9ffc23fe-8cc5-4d63-bc98-fba3c171561b\") " pod="openshift-ingress-canary/ingress-canary-4pvfb" Jul 29 08:48:42.975091 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:42.975074 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvvzm\" (UniqueName: \"kubernetes.io/projected/bc60cc72-c068-4d53-a2ad-4141c76472a5-kube-api-access-wvvzm\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:43.094600 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:43.094563 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4pvfb" Jul 29 08:48:43.226016 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:43.225836 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4pvfb"] Jul 29 08:48:43.229427 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:43.229402 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffc23fe_8cc5_4d63_bc98_fba3c171561b.slice/crio-656aa4c4b15fba20de2c49825a78aaf2af3da6dd6c10abbc5ebac5ed9f8e234e WatchSource:0}: Error finding container 656aa4c4b15fba20de2c49825a78aaf2af3da6dd6c10abbc5ebac5ed9f8e234e: Status 404 returned error can't find the container with id 656aa4c4b15fba20de2c49825a78aaf2af3da6dd6c10abbc5ebac5ed9f8e234e Jul 29 08:48:43.469146 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:43.469106 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:43.469744 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:43.469259 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:48:43.469744 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:43.469341 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls podName:bc60cc72-c068-4d53-a2ad-4141c76472a5 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:44.46932438 +0000 UTC m=+37.593425157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls") pod "dns-default-7ff2z" (UID: "bc60cc72-c068-4d53-a2ad-4141c76472a5") : secret "dns-default-metrics-tls" not found Jul 29 08:48:43.532116 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:43.532070 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4pvfb" event={"ID":"9ffc23fe-8cc5-4d63-bc98-fba3c171561b","Type":"ContainerStarted","Data":"656aa4c4b15fba20de2c49825a78aaf2af3da6dd6c10abbc5ebac5ed9f8e234e"} Jul 29 08:48:43.534639 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:43.534614 2563 generic.go:356] "Generic (PLEG): container finished" podID="510492e8-2257-458d-aff3-b69dd243e9d5" containerID="01dcc91e43dac7fc6543cb3fdb41f46edd7a79cab9d8a7fc6e143ff1a3bb80d8" exitCode=0 Jul 29 08:48:43.534764 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:43.534650 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvlsg" event={"ID":"510492e8-2257-458d-aff3-b69dd243e9d5","Type":"ContainerDied","Data":"01dcc91e43dac7fc6543cb3fdb41f46edd7a79cab9d8a7fc6e143ff1a3bb80d8"} Jul 29 08:48:44.356624 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:44.356588 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:48:44.356624 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:44.356614 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:48:44.356861 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:44.356600 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:48:44.361509 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:44.361315 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dmmnd\"" Jul 29 08:48:44.361509 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:44.361337 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4hkhb\"" Jul 29 08:48:44.361509 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:44.361356 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 08:48:44.361509 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:44.361411 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 08:48:44.361851 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:44.361616 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 08:48:44.361851 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:44.361652 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 08:48:44.476740 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:44.476706 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:44.477185 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:44.476811 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:48:44.477185 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:44.476905 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls podName:bc60cc72-c068-4d53-a2ad-4141c76472a5 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:46.476879965 +0000 UTC m=+39.600980746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls") pod "dns-default-7ff2z" (UID: "bc60cc72-c068-4d53-a2ad-4141c76472a5") : secret "dns-default-metrics-tls" not found Jul 29 08:48:44.541107 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:44.541067 2563 generic.go:356] "Generic (PLEG): container finished" podID="510492e8-2257-458d-aff3-b69dd243e9d5" containerID="57cc539ced53a9aebf4688a8707f6540c10b949ed4a203c92248e93e160acb0c" exitCode=0 Jul 29 08:48:44.541268 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:44.541121 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvlsg" event={"ID":"510492e8-2257-458d-aff3-b69dd243e9d5","Type":"ContainerDied","Data":"57cc539ced53a9aebf4688a8707f6540c10b949ed4a203c92248e93e160acb0c"} Jul 29 08:48:45.546228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:45.546193 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hvlsg" event={"ID":"510492e8-2257-458d-aff3-b69dd243e9d5","Type":"ContainerStarted","Data":"df84ab34aa5f8d4c2996894b5841f2bcb9077ee9468265276330cb7989ec69a3"} Jul 29 08:48:45.569135 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:45.569087 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hvlsg" podStartSLOduration=7.029243533 podStartE2EDuration="37.56907215s" totalImagesPullingTime="30.539828617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:48:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:45.567441641 +0000 UTC m=+38.691542444" watchObservedRunningTime="2026-07-29 08:48:45.56907215 +0000 UTC m=+38.693172941" Jul 29 08:48:46.492654 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:46.492438 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:46.492814 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:46.492593 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:48:46.492814 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:46.492730 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls podName:bc60cc72-c068-4d53-a2ad-4141c76472a5 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:50.492711441 +0000 UTC m=+43.616812238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls") pod "dns-default-7ff2z" (UID: "bc60cc72-c068-4d53-a2ad-4141c76472a5") : secret "dns-default-metrics-tls" not found Jul 29 08:48:46.550398 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:46.550344 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4pvfb" event={"ID":"9ffc23fe-8cc5-4d63-bc98-fba3c171561b","Type":"ContainerStarted","Data":"ec240f9b9096c5b3fbbf49b9719609b2fb566a301a7ec51242eeace9c10c3e6f"} Jul 29 08:48:46.566884 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:46.566837 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4pvfb" podStartSLOduration=2.278320609 podStartE2EDuration="4.566821045s" totalImagesPullingTime="2.288500436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:46.565523941 +0000 UTC m=+39.689624741" watchObservedRunningTime="2026-07-29 08:48:46.566821045 +0000 UTC m=+39.690921844" Jul 29 08:48:50.006675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.006635 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9"] Jul 29 08:48:50.039822 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.039783 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9"] Jul 29 08:48:50.039973 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.039902 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" Jul 29 08:48:50.042634 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.042604 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 29 08:48:50.042634 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.042618 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 29 08:48:50.042821 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.042735 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 29 08:48:50.043839 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.043822 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 29 08:48:50.043954 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.043866 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4s9cg\"" Jul 29 08:48:50.214558 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.214524 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf"] Jul 29 08:48:50.219837 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.219809 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e723fb2-089d-411b-9a4b-07d8d1b4cfa1-serving-cert\") pod \"service-ca-operator-5d6c75f84f-8s2j9\" (UID: \"1e723fb2-089d-411b-9a4b-07d8d1b4cfa1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" Jul 29 08:48:50.219970 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.219861 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e723fb2-089d-411b-9a4b-07d8d1b4cfa1-config\") pod \"service-ca-operator-5d6c75f84f-8s2j9\" (UID: \"1e723fb2-089d-411b-9a4b-07d8d1b4cfa1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" Jul 29 08:48:50.219970 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.219882 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtwh\" (UniqueName: \"kubernetes.io/projected/1e723fb2-089d-411b-9a4b-07d8d1b4cfa1-kube-api-access-lbtwh\") pod \"service-ca-operator-5d6c75f84f-8s2j9\" (UID: \"1e723fb2-089d-411b-9a4b-07d8d1b4cfa1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" Jul 29 08:48:50.237564 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.237542 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-798649959c-9mnkr"] Jul 29 08:48:50.237691 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.237674 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" Jul 29 08:48:50.240446 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.240427 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 29 08:48:50.240555 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.240465 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wn8r6\"" Jul 29 08:48:50.240555 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.240478 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 29 08:48:50.240645 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.240552 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 29 08:48:50.261586 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.261527 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf"] Jul 29 08:48:50.261586 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.261567 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-798649959c-9mnkr"] Jul 29 08:48:50.261709 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.261643 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.264080 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.264063 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 08:48:50.264265 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.264250 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 08:48:50.264407 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.264393 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 08:48:50.264527 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.264512 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-t2qgg\"" Jul 29 08:48:50.272823 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.272799 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 08:48:50.320564 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.320533 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e723fb2-089d-411b-9a4b-07d8d1b4cfa1-serving-cert\") pod \"service-ca-operator-5d6c75f84f-8s2j9\" (UID: \"1e723fb2-089d-411b-9a4b-07d8d1b4cfa1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" Jul 29 08:48:50.320656 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.320585 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e723fb2-089d-411b-9a4b-07d8d1b4cfa1-config\") pod \"service-ca-operator-5d6c75f84f-8s2j9\" (UID: \"1e723fb2-089d-411b-9a4b-07d8d1b4cfa1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" Jul 29 08:48:50.320656 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.320615 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtwh\" (UniqueName: \"kubernetes.io/projected/1e723fb2-089d-411b-9a4b-07d8d1b4cfa1-kube-api-access-lbtwh\") pod \"service-ca-operator-5d6c75f84f-8s2j9\" (UID: \"1e723fb2-089d-411b-9a4b-07d8d1b4cfa1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" Jul 29 08:48:50.321226 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.321208 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e723fb2-089d-411b-9a4b-07d8d1b4cfa1-config\") pod \"service-ca-operator-5d6c75f84f-8s2j9\" (UID: \"1e723fb2-089d-411b-9a4b-07d8d1b4cfa1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" Jul 29 08:48:50.324392 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.324358 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e723fb2-089d-411b-9a4b-07d8d1b4cfa1-serving-cert\") pod \"service-ca-operator-5d6c75f84f-8s2j9\" (UID: \"1e723fb2-089d-411b-9a4b-07d8d1b4cfa1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" Jul 29 08:48:50.328684 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.328664 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtwh\" (UniqueName: \"kubernetes.io/projected/1e723fb2-089d-411b-9a4b-07d8d1b4cfa1-kube-api-access-lbtwh\") pod \"service-ca-operator-5d6c75f84f-8s2j9\" (UID: \"1e723fb2-089d-411b-9a4b-07d8d1b4cfa1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" Jul 29 08:48:50.348595 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.348562 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" Jul 29 08:48:50.421883 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.421853 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.422003 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.421892 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0b774629-0437-451c-9d63-777851e043e2-ca-trust-extracted\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.422003 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.421919 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lm5zf\" (UID: \"62796450-05af-4a92-9144-fd25121a04e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" Jul 29 08:48:50.422124 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.422006 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qjrz\" (UniqueName: \"kubernetes.io/projected/62796450-05af-4a92-9144-fd25121a04e7-kube-api-access-5qjrz\") pod \"cluster-samples-operator-6748489656-lm5zf\" (UID: \"62796450-05af-4a92-9144-fd25121a04e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" Jul 29 08:48:50.422124 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.422056 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0b774629-0437-451c-9d63-777851e043e2-image-registry-private-configuration\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.422124 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.422088 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-bound-sa-token\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.422269 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.422129 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0b774629-0437-451c-9d63-777851e043e2-installation-pull-secrets\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.422269 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.422214 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5tvn\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-kube-api-access-v5tvn\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.422340 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.422302 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0b774629-0437-451c-9d63-777851e043e2-registry-certificates\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.422401 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.422381 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b774629-0437-451c-9d63-777851e043e2-trusted-ca\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.493732 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.493701 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9"] Jul 29 08:48:50.496516 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:48:50.496490 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e723fb2_089d_411b_9a4b_07d8d1b4cfa1.slice/crio-0eb5953e750626eaa20bffb69366fd009fa38b18915f59e6f5a9f634263ce33f WatchSource:0}: Error finding container 0eb5953e750626eaa20bffb69366fd009fa38b18915f59e6f5a9f634263ce33f: Status 404 returned error can't find the container with id 0eb5953e750626eaa20bffb69366fd009fa38b18915f59e6f5a9f634263ce33f Jul 29 08:48:50.522822 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.522763 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b774629-0437-451c-9d63-777851e043e2-trusted-ca\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.522822 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.522803 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:50.522945 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.522823 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.522945 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.522839 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0b774629-0437-451c-9d63-777851e043e2-ca-trust-extracted\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.522945 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.522855 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lm5zf\" (UID: \"62796450-05af-4a92-9144-fd25121a04e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" Jul 29 08:48:50.522945 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.522874 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qjrz\" (UniqueName: \"kubernetes.io/projected/62796450-05af-4a92-9144-fd25121a04e7-kube-api-access-5qjrz\") pod \"cluster-samples-operator-6748489656-lm5zf\" (UID: \"62796450-05af-4a92-9144-fd25121a04e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" Jul 29 08:48:50.522945 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:50.522931 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:48:50.523160 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:50.522977 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 08:48:50.523160 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:50.522936 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 08:48:50.523160 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:50.523012 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls podName:bc60cc72-c068-4d53-a2ad-4141c76472a5 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:58.522976145 +0000 UTC m=+51.647076922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls") pod "dns-default-7ff2z" (UID: "bc60cc72-c068-4d53-a2ad-4141c76472a5") : secret "dns-default-metrics-tls" not found Jul 29 08:48:50.523160 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:50.523011 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-798649959c-9mnkr: secret "image-registry-tls" not found Jul 29 08:48:50.523160 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:50.523044 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls podName:62796450-05af-4a92-9144-fd25121a04e7 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:51.023027006 +0000 UTC m=+44.147127798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls") pod "cluster-samples-operator-6748489656-lm5zf" (UID: "62796450-05af-4a92-9144-fd25121a04e7") : secret "samples-operator-tls" not found Jul 29 08:48:50.523160 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.523038 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0b774629-0437-451c-9d63-777851e043e2-image-registry-private-configuration\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.523160 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:50.523089 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls podName:0b774629-0437-451c-9d63-777851e043e2 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:51.02307006 +0000 UTC m=+44.147170850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls") pod "image-registry-798649959c-9mnkr" (UID: "0b774629-0437-451c-9d63-777851e043e2") : secret "image-registry-tls" not found Jul 29 08:48:50.523160 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.523116 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-bound-sa-token\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.523160 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.523159 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0b774629-0437-451c-9d63-777851e043e2-installation-pull-secrets\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.523617 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.523245 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5tvn\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-kube-api-access-v5tvn\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.523617 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.523261 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0b774629-0437-451c-9d63-777851e043e2-ca-trust-extracted\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.523617 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.523299 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0b774629-0437-451c-9d63-777851e043e2-registry-certificates\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.523874 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.523855 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0b774629-0437-451c-9d63-777851e043e2-registry-certificates\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.523963 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.523944 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b774629-0437-451c-9d63-777851e043e2-trusted-ca\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.525274 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.525255 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0b774629-0437-451c-9d63-777851e043e2-image-registry-private-configuration\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.525379 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.525349 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0b774629-0437-451c-9d63-777851e043e2-installation-pull-secrets\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.531749 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.531722 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-bound-sa-token\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.531819 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.531783 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5tvn\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-kube-api-access-v5tvn\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:50.531932 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.531913 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qjrz\" (UniqueName: \"kubernetes.io/projected/62796450-05af-4a92-9144-fd25121a04e7-kube-api-access-5qjrz\") pod \"cluster-samples-operator-6748489656-lm5zf\" (UID: \"62796450-05af-4a92-9144-fd25121a04e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" Jul 29 08:48:50.560732 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:50.560701 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" event={"ID":"1e723fb2-089d-411b-9a4b-07d8d1b4cfa1","Type":"ContainerStarted","Data":"0eb5953e750626eaa20bffb69366fd009fa38b18915f59e6f5a9f634263ce33f"} Jul 29 08:48:51.026888 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:51.026845 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:51.026888 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:51.026886 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lm5zf\" (UID: \"62796450-05af-4a92-9144-fd25121a04e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" Jul 29 08:48:51.027309 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:51.026997 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 08:48:51.027309 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:51.027011 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 08:48:51.027309 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:51.027016 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-798649959c-9mnkr: secret "image-registry-tls" not found Jul 29 08:48:51.027309 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:51.027067 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls podName:0b774629-0437-451c-9d63-777851e043e2 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:52.027052985 +0000 UTC m=+45.151153762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls") pod "image-registry-798649959c-9mnkr" (UID: "0b774629-0437-451c-9d63-777851e043e2") : secret "image-registry-tls" not found Jul 29 08:48:51.027309 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:51.027080 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls podName:62796450-05af-4a92-9144-fd25121a04e7 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:52.027073666 +0000 UTC m=+45.151174443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls") pod "cluster-samples-operator-6748489656-lm5zf" (UID: "62796450-05af-4a92-9144-fd25121a04e7") : secret "samples-operator-tls" not found Jul 29 08:48:52.036953 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:52.036916 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:52.037446 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:52.036964 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lm5zf\" (UID: \"62796450-05af-4a92-9144-fd25121a04e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" Jul 29 08:48:52.037446 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:52.037095 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 08:48:52.037446 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:52.037117 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-798649959c-9mnkr: secret "image-registry-tls" not found Jul 29 08:48:52.037446 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:52.037159 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 08:48:52.037446 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:52.037177 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls podName:0b774629-0437-451c-9d63-777851e043e2 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:54.037162235 +0000 UTC m=+47.161263012 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls") pod "image-registry-798649959c-9mnkr" (UID: "0b774629-0437-451c-9d63-777851e043e2") : secret "image-registry-tls" not found Jul 29 08:48:52.037446 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:52.037222 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls podName:62796450-05af-4a92-9144-fd25121a04e7 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:54.037205309 +0000 UTC m=+47.161306088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls") pod "cluster-samples-operator-6748489656-lm5zf" (UID: "62796450-05af-4a92-9144-fd25121a04e7") : secret "samples-operator-tls" not found Jul 29 08:48:53.569692 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:53.569656 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" event={"ID":"1e723fb2-089d-411b-9a4b-07d8d1b4cfa1","Type":"ContainerStarted","Data":"e39f415371a29812ac31c6212bda2074da2c10cdebd22df36c6381f106f560bc"} Jul 29 08:48:53.591752 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:53.591700 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" podStartSLOduration=1.726096068 podStartE2EDuration="4.591679982s" totalImagesPullingTime="2.865583914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:48:53.590139119 +0000 UTC m=+46.714239929" watchObservedRunningTime="2026-07-29 08:48:53.591679982 +0000 UTC m=+46.715780758" Jul 29 08:48:54.054431 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:54.054399 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:54.054431 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:54.054432 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lm5zf\" (UID: \"62796450-05af-4a92-9144-fd25121a04e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" Jul 29 08:48:54.054701 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:54.054577 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 08:48:54.054701 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:54.054603 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-798649959c-9mnkr: secret "image-registry-tls" not found Jul 29 08:48:54.054701 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:54.054672 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls podName:0b774629-0437-451c-9d63-777851e043e2 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:58.05465105 +0000 UTC m=+51.178751847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls") pod "image-registry-798649959c-9mnkr" (UID: "0b774629-0437-451c-9d63-777851e043e2") : secret "image-registry-tls" not found Jul 29 08:48:54.054701 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:54.054604 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 08:48:54.054897 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:54.054717 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls podName:62796450-05af-4a92-9144-fd25121a04e7 nodeName:}" failed. No retries permitted until 2026-07-29 08:48:58.054706621 +0000 UTC m=+51.178807399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls") pod "cluster-samples-operator-6748489656-lm5zf" (UID: "62796450-05af-4a92-9144-fd25121a04e7") : secret "samples-operator-tls" not found Jul 29 08:48:56.184880 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.184844 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nfxfj"] Jul 29 08:48:56.207643 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.206675 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nfxfj"] Jul 29 08:48:56.207643 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.206805 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.210178 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.210149 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 08:48:56.210178 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.210174 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jw2tp\"" Jul 29 08:48:56.210339 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.210293 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 08:48:56.210625 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.210605 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 08:48:56.210972 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.210953 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 08:48:56.270200 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.270164 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-data-volume\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.270200 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.270200 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.270461 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.270261 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg8l5\" (UniqueName: \"kubernetes.io/projected/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-api-access-sg8l5\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.270461 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.270292 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.270461 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.270395 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-crio-socket\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.370912 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.370878 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg8l5\" (UniqueName: \"kubernetes.io/projected/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-api-access-sg8l5\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.370912 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.370920 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.371160 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.370950 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-crio-socket\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.371160 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:56.371020 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 08:48:56.371160 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.371100 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-crio-socket\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.371160 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.371110 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-data-volume\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.371160 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.371135 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.371382 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:56.371186 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm podName:6c3c1c04-de29-4b84-bbb3-f6e5107979dd nodeName:}" failed. No retries permitted until 2026-07-29 08:48:56.871163441 +0000 UTC m=+49.995264229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nfxfj" (UID: "6c3c1c04-de29-4b84-bbb3-f6e5107979dd") : configmap "kube-rbac-proxy" not found Jul 29 08:48:56.371382 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:56.371229 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:48:56.371382 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:56.371284 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls podName:6c3c1c04-de29-4b84-bbb3-f6e5107979dd nodeName:}" failed. No retries permitted until 2026-07-29 08:48:56.87126698 +0000 UTC m=+49.995367757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nfxfj" (UID: "6c3c1c04-de29-4b84-bbb3-f6e5107979dd") : secret "insights-runtime-extractor-tls" not found Jul 29 08:48:56.371491 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.371429 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-data-volume\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.379533 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.379507 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg8l5\" (UniqueName: \"kubernetes.io/projected/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-api-access-sg8l5\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.875011 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.874976 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.875179 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:56.875049 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:56.875179 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:56.875134 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:48:56.875179 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:56.875166 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 08:48:56.875281 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:56.875202 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls podName:6c3c1c04-de29-4b84-bbb3-f6e5107979dd nodeName:}" failed. No retries permitted until 2026-07-29 08:48:57.875187264 +0000 UTC m=+50.999288042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nfxfj" (UID: "6c3c1c04-de29-4b84-bbb3-f6e5107979dd") : secret "insights-runtime-extractor-tls" not found Jul 29 08:48:56.875281 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:56.875229 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm podName:6c3c1c04-de29-4b84-bbb3-f6e5107979dd nodeName:}" failed. No retries permitted until 2026-07-29 08:48:57.875217043 +0000 UTC m=+50.999317824 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nfxfj" (UID: "6c3c1c04-de29-4b84-bbb3-f6e5107979dd") : configmap "kube-rbac-proxy" not found Jul 29 08:48:57.882113 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:57.882076 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:57.882583 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:57.882178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:57.882583 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:57.882239 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:48:57.882583 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:57.882296 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 08:48:57.882583 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:57.882302 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls podName:6c3c1c04-de29-4b84-bbb3-f6e5107979dd nodeName:}" failed. No retries permitted until 2026-07-29 08:48:59.882287494 +0000 UTC m=+53.006388311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nfxfj" (UID: "6c3c1c04-de29-4b84-bbb3-f6e5107979dd") : secret "insights-runtime-extractor-tls" not found Jul 29 08:48:57.882583 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:57.882399 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm podName:6c3c1c04-de29-4b84-bbb3-f6e5107979dd nodeName:}" failed. No retries permitted until 2026-07-29 08:48:59.882381917 +0000 UTC m=+53.006482707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nfxfj" (UID: "6c3c1c04-de29-4b84-bbb3-f6e5107979dd") : configmap "kube-rbac-proxy" not found Jul 29 08:48:58.087644 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:58.087609 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:48:58.087812 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:58.087649 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lm5zf\" (UID: \"62796450-05af-4a92-9144-fd25121a04e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" Jul 29 08:48:58.087812 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:58.087735 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 08:48:58.087812 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:58.087757 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-798649959c-9mnkr: secret "image-registry-tls" not found Jul 29 08:48:58.087812 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:58.087815 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls podName:0b774629-0437-451c-9d63-777851e043e2 nodeName:}" failed. No retries permitted until 2026-07-29 08:49:06.087798228 +0000 UTC m=+59.211899025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls") pod "image-registry-798649959c-9mnkr" (UID: "0b774629-0437-451c-9d63-777851e043e2") : secret "image-registry-tls" not found Jul 29 08:48:58.087956 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:58.087813 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 08:48:58.087956 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:58.087841 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls podName:62796450-05af-4a92-9144-fd25121a04e7 nodeName:}" failed. No retries permitted until 2026-07-29 08:49:06.087835591 +0000 UTC m=+59.211936367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls") pod "cluster-samples-operator-6748489656-lm5zf" (UID: "62796450-05af-4a92-9144-fd25121a04e7") : secret "samples-operator-tls" not found Jul 29 08:48:58.591149 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:58.591113 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:48:58.591314 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:58.591220 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 08:48:58.591314 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:58.591267 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls podName:bc60cc72-c068-4d53-a2ad-4141c76472a5 nodeName:}" failed. No retries permitted until 2026-07-29 08:49:14.591253829 +0000 UTC m=+67.715354606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls") pod "dns-default-7ff2z" (UID: "bc60cc72-c068-4d53-a2ad-4141c76472a5") : secret "dns-default-metrics-tls" not found Jul 29 08:48:58.948314 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:58.948285 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hmb59_86239d8c-0cc4-4be4-b21a-e31d5139934c/dns-node-resolver/0.log" Jul 29 08:48:59.901609 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:59.901570 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:59.901772 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:59.901647 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:48:59.901772 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:59.901732 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:48:59.901772 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:59.901751 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 08:48:59.901911 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:59.901802 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls podName:6c3c1c04-de29-4b84-bbb3-f6e5107979dd nodeName:}" failed. No retries permitted until 2026-07-29 08:49:03.901785718 +0000 UTC m=+57.025886495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nfxfj" (UID: "6c3c1c04-de29-4b84-bbb3-f6e5107979dd") : secret "insights-runtime-extractor-tls" not found Jul 29 08:48:59.901911 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:48:59.901819 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm podName:6c3c1c04-de29-4b84-bbb3-f6e5107979dd nodeName:}" failed. No retries permitted until 2026-07-29 08:49:03.901811176 +0000 UTC m=+57.025911953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nfxfj" (UID: "6c3c1c04-de29-4b84-bbb3-f6e5107979dd") : configmap "kube-rbac-proxy" not found Jul 29 08:48:59.952453 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:48:59.952429 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cpx2c_1422c6a2-0287-4e57-8379-b18d6fcef17e/node-ca/0.log" Jul 29 08:49:00.547349 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:00.547315 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4pvfb_9ffc23fe-8cc5-4d63-bc98-fba3c171561b/serve-healthcheck-canary/0.log" Jul 29 08:49:03.933603 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:03.933572 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:49:03.934066 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:03.933661 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:49:03.934066 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:49:03.933709 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 08:49:03.934066 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:49:03.933774 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm podName:6c3c1c04-de29-4b84-bbb3-f6e5107979dd nodeName:}" failed. No retries permitted until 2026-07-29 08:49:11.933758145 +0000 UTC m=+65.057858935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nfxfj" (UID: "6c3c1c04-de29-4b84-bbb3-f6e5107979dd") : configmap "kube-rbac-proxy" not found Jul 29 08:49:03.934066 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:49:03.933806 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 08:49:03.934066 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:49:03.933866 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls podName:6c3c1c04-de29-4b84-bbb3-f6e5107979dd nodeName:}" failed. No retries permitted until 2026-07-29 08:49:11.933852363 +0000 UTC m=+65.057953151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nfxfj" (UID: "6c3c1c04-de29-4b84-bbb3-f6e5107979dd") : secret "insights-runtime-extractor-tls" not found Jul 29 08:49:06.151249 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:06.151205 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:49:06.151249 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:06.151248 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lm5zf\" (UID: \"62796450-05af-4a92-9144-fd25121a04e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" Jul 29 08:49:06.153632 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:06.153599 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls\") pod \"image-registry-798649959c-9mnkr\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:49:06.153773 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:06.153698 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/62796450-05af-4a92-9144-fd25121a04e7-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lm5zf\" (UID: \"62796450-05af-4a92-9144-fd25121a04e7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" Jul 29 08:49:06.172722 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:06.172695 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:49:06.291848 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:06.291818 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-798649959c-9mnkr"] Jul 29 08:49:06.446194 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:06.446157 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" Jul 29 08:49:06.569437 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:06.569396 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf"] Jul 29 08:49:06.600859 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:06.600829 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-798649959c-9mnkr" event={"ID":"0b774629-0437-451c-9d63-777851e043e2","Type":"ContainerStarted","Data":"1d970c7156959030bc34fb2ad2e93f221e8ddea8d8efc4e43d349c9abc20fa28"} Jul 29 08:49:06.601008 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:06.600883 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-798649959c-9mnkr" event={"ID":"0b774629-0437-451c-9d63-777851e043e2","Type":"ContainerStarted","Data":"fdcdcc0513bff85db0a081365764bafa7ef09b9f133a5dae1862826e4b770d91"} Jul 29 08:49:06.601008 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:06.600971 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:49:07.531783 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:07.531754 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s7zc5" Jul 29 08:49:07.559495 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:07.559290 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-798649959c-9mnkr" podStartSLOduration=17.559271025 podStartE2EDuration="17.559271025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:06.621766556 +0000 UTC m=+59.745867355" watchObservedRunningTime="2026-07-29 08:49:07.559271025 +0000 UTC m=+60.683371825" Jul 29 08:49:07.605146 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:07.605065 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" event={"ID":"62796450-05af-4a92-9144-fd25121a04e7","Type":"ContainerStarted","Data":"dfca1ebe4b59fce6d9d1b0d6460bb44184c6d382bd1fe8aa5b72ec4f01893c96"} Jul 29 08:49:09.611657 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:09.611613 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" event={"ID":"62796450-05af-4a92-9144-fd25121a04e7","Type":"ContainerStarted","Data":"da4b779ee253230dd4c5377bf57e42c1a3b3f2ad1577e51903f7b6e89fb5c74b"} Jul 29 08:49:09.612045 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:09.611702 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" event={"ID":"62796450-05af-4a92-9144-fd25121a04e7","Type":"ContainerStarted","Data":"28027e6dcdd61cea7d3ea27d86a4947962e80ffaa2b1f022df3d9329f7ab17a2"} Jul 29 08:49:09.631343 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:09.631296 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lm5zf" podStartSLOduration=17.685879683 podStartE2EDuration="19.63128075s" totalImagesPullingTime="1.945401067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:09.630165102 +0000 UTC m=+62.754265903" watchObservedRunningTime="2026-07-29 08:49:09.63128075 +0000 UTC m=+62.755381590" Jul 29 08:49:11.997418 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:11.997380 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:49:11.997807 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:11.997450 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:49:11.997903 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:11.997881 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:49:11.999593 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:11.999569 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6c3c1c04-de29-4b84-bbb3-f6e5107979dd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nfxfj\" (UID: \"6c3c1c04-de29-4b84-bbb3-f6e5107979dd\") " pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:49:12.120803 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:12.120768 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jw2tp\"" Jul 29 08:49:12.128206 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:12.128163 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nfxfj" Jul 29 08:49:12.252738 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:12.252655 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nfxfj"] Jul 29 08:49:12.255581 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:49:12.255553 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3c1c04_de29_4b84_bbb3_f6e5107979dd.slice/crio-21e89c123c1ba98d10e4248d205afbd56c72534cf3a0121789f28ea75ce40a7c WatchSource:0}: Error finding container 21e89c123c1ba98d10e4248d205afbd56c72534cf3a0121789f28ea75ce40a7c: Status 404 returned error can't find the container with id 21e89c123c1ba98d10e4248d205afbd56c72534cf3a0121789f28ea75ce40a7c Jul 29 08:49:12.619704 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:12.619618 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nfxfj" event={"ID":"6c3c1c04-de29-4b84-bbb3-f6e5107979dd","Type":"ContainerStarted","Data":"21e89c123c1ba98d10e4248d205afbd56c72534cf3a0121789f28ea75ce40a7c"} Jul 29 08:49:13.105553 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.105509 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:49:13.106050 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.105598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs\") pod \"network-metrics-daemon-9dgqn\" (UID: \"d5fac53f-30ee-496f-9131-c819ae8e4963\") " pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:49:13.108711 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.108684 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 08:49:13.108838 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.108773 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 08:49:13.119010 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.118958 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9014122b-1078-4e35-b53c-234e93bbd054-original-pull-secret\") pod \"global-pull-secret-syncer-z6kw7\" (UID: \"9014122b-1078-4e35-b53c-234e93bbd054\") " pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:49:13.119010 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.118996 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5fac53f-30ee-496f-9131-c819ae8e4963-metrics-certs\") pod \"network-metrics-daemon-9dgqn\" (UID: \"d5fac53f-30ee-496f-9131-c819ae8e4963\") " pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:49:13.171992 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.171956 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dmmnd\"" Jul 29 08:49:13.176801 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.176558 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z6kw7" Jul 29 08:49:13.179561 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.179535 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9dgqn" Jul 29 08:49:13.206965 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.206922 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclsb\" (UniqueName: \"kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb\") pod \"network-check-target-rwtwp\" (UID: \"3e7aa462-6508-4fa0-ab6d-43a5670db158\") " pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:49:13.210132 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.210060 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 08:49:13.220999 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.220780 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 08:49:13.231267 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.231205 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pclsb\" (UniqueName: \"kubernetes.io/projected/3e7aa462-6508-4fa0-ab6d-43a5670db158-kube-api-access-pclsb\") pod \"network-check-target-rwtwp\" (UID: \"3e7aa462-6508-4fa0-ab6d-43a5670db158\") " pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:49:13.337179 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.337150 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z6kw7"] Jul 29 08:49:13.341265 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:49:13.341232 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9014122b_1078_4e35_b53c_234e93bbd054.slice/crio-2242484cff30a1bcbb713e18c8d9f1fc894190fdde175456dee55ecc1a8f8539 WatchSource:0}: Error finding container 2242484cff30a1bcbb713e18c8d9f1fc894190fdde175456dee55ecc1a8f8539: Status 404 returned error can't find the container with id 2242484cff30a1bcbb713e18c8d9f1fc894190fdde175456dee55ecc1a8f8539 Jul 29 08:49:13.355960 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.355855 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9dgqn"] Jul 29 08:49:13.359158 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:49:13.359104 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fac53f_30ee_496f_9131_c819ae8e4963.slice/crio-f9381b92248b37e0358624636dc3ef6423f2b5e5b7740630ffbb508a21006813 WatchSource:0}: Error finding container f9381b92248b37e0358624636dc3ef6423f2b5e5b7740630ffbb508a21006813: Status 404 returned error can't find the container with id f9381b92248b37e0358624636dc3ef6423f2b5e5b7740630ffbb508a21006813 Jul 29 08:49:13.485697 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.485668 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4hkhb\"" Jul 29 08:49:13.494002 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.493978 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:49:13.623700 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.623612 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z6kw7" event={"ID":"9014122b-1078-4e35-b53c-234e93bbd054","Type":"ContainerStarted","Data":"2242484cff30a1bcbb713e18c8d9f1fc894190fdde175456dee55ecc1a8f8539"} Jul 29 08:49:13.625151 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.625113 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9dgqn" event={"ID":"d5fac53f-30ee-496f-9131-c819ae8e4963","Type":"ContainerStarted","Data":"f9381b92248b37e0358624636dc3ef6423f2b5e5b7740630ffbb508a21006813"} Jul 29 08:49:13.629474 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:13.629405 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rwtwp"] Jul 29 08:49:13.634156 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:49:13.634128 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7aa462_6508_4fa0_ab6d_43a5670db158.slice/crio-adeef69b1045ca483fb7a42bc60936afc6466976fb07c467cda76fa06a48196d WatchSource:0}: Error finding container adeef69b1045ca483fb7a42bc60936afc6466976fb07c467cda76fa06a48196d: Status 404 returned error can't find the container with id adeef69b1045ca483fb7a42bc60936afc6466976fb07c467cda76fa06a48196d Jul 29 08:49:14.618824 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:14.618792 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:49:14.621455 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:14.621429 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc60cc72-c068-4d53-a2ad-4141c76472a5-metrics-tls\") pod \"dns-default-7ff2z\" (UID: \"bc60cc72-c068-4d53-a2ad-4141c76472a5\") " pod="openshift-dns/dns-default-7ff2z" Jul 29 08:49:14.630036 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:14.630004 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rwtwp" event={"ID":"3e7aa462-6508-4fa0-ab6d-43a5670db158","Type":"ContainerStarted","Data":"adeef69b1045ca483fb7a42bc60936afc6466976fb07c467cda76fa06a48196d"} Jul 29 08:49:14.892169 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:14.891896 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-68zl6\"" Jul 29 08:49:14.899844 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:14.899802 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7ff2z" Jul 29 08:49:17.889509 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:17.889474 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg"] Jul 29 08:49:17.892598 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:17.892578 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" Jul 29 08:49:17.899390 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:17.899350 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 29 08:49:17.899571 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:17.899549 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 29 08:49:17.901082 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:17.900516 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 29 08:49:17.901082 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:17.900865 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 29 08:49:17.914824 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:17.914800 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg"] Jul 29 08:49:17.934646 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:17.934615 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-798649959c-9mnkr"] Jul 29 08:49:17.944650 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:17.944624 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/78f4aa74-7387-47a8-83bc-5bff5887fb2e-klusterlet-config\") pod \"klusterlet-addon-workmgr-d9b57869b-hlkpg\" (UID: \"78f4aa74-7387-47a8-83bc-5bff5887fb2e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" Jul 29 08:49:17.944785 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:17.944664 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78f4aa74-7387-47a8-83bc-5bff5887fb2e-tmp\") pod \"klusterlet-addon-workmgr-d9b57869b-hlkpg\" (UID: \"78f4aa74-7387-47a8-83bc-5bff5887fb2e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" Jul 29 08:49:17.944785 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:17.944720 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvflp\" (UniqueName: \"kubernetes.io/projected/78f4aa74-7387-47a8-83bc-5bff5887fb2e-kube-api-access-pvflp\") pod \"klusterlet-addon-workmgr-d9b57869b-hlkpg\" (UID: \"78f4aa74-7387-47a8-83bc-5bff5887fb2e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" Jul 29 08:49:17.987985 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:17.987952 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-qks4q"] Jul 29 08:49:17.989855 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:17.989833 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qks4q" Jul 29 08:49:18.006485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.006462 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kchc2\"" Jul 29 08:49:18.006969 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.006950 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 29 08:49:18.007043 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.006950 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 29 08:49:18.019511 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.019483 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-qks4q"] Jul 29 08:49:18.045961 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.045924 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvflp\" (UniqueName: \"kubernetes.io/projected/78f4aa74-7387-47a8-83bc-5bff5887fb2e-kube-api-access-pvflp\") pod \"klusterlet-addon-workmgr-d9b57869b-hlkpg\" (UID: \"78f4aa74-7387-47a8-83bc-5bff5887fb2e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" Jul 29 08:49:18.046113 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.045991 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/78f4aa74-7387-47a8-83bc-5bff5887fb2e-klusterlet-config\") pod \"klusterlet-addon-workmgr-d9b57869b-hlkpg\" (UID: \"78f4aa74-7387-47a8-83bc-5bff5887fb2e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" Jul 29 08:49:18.046180 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.046137 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41e45e49-9c61-43f9-86a8-1a46d12a4544-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qks4q\" (UID: \"41e45e49-9c61-43f9-86a8-1a46d12a4544\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qks4q" Jul 29 08:49:18.046234 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.046182 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78f4aa74-7387-47a8-83bc-5bff5887fb2e-tmp\") pod \"klusterlet-addon-workmgr-d9b57869b-hlkpg\" (UID: \"78f4aa74-7387-47a8-83bc-5bff5887fb2e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" Jul 29 08:49:18.046234 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.046213 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/41e45e49-9c61-43f9-86a8-1a46d12a4544-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-qks4q\" (UID: \"41e45e49-9c61-43f9-86a8-1a46d12a4544\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qks4q" Jul 29 08:49:18.046649 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.046626 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78f4aa74-7387-47a8-83bc-5bff5887fb2e-tmp\") pod \"klusterlet-addon-workmgr-d9b57869b-hlkpg\" (UID: \"78f4aa74-7387-47a8-83bc-5bff5887fb2e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" Jul 29 08:49:18.048773 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.048753 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/78f4aa74-7387-47a8-83bc-5bff5887fb2e-klusterlet-config\") pod \"klusterlet-addon-workmgr-d9b57869b-hlkpg\" (UID: \"78f4aa74-7387-47a8-83bc-5bff5887fb2e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" Jul 29 08:49:18.072915 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.072889 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvflp\" (UniqueName: \"kubernetes.io/projected/78f4aa74-7387-47a8-83bc-5bff5887fb2e-kube-api-access-pvflp\") pod \"klusterlet-addon-workmgr-d9b57869b-hlkpg\" (UID: \"78f4aa74-7387-47a8-83bc-5bff5887fb2e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" Jul 29 08:49:18.104080 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.104052 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-sn9p9"] Jul 29 08:49:18.106153 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.106128 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-sn9p9" Jul 29 08:49:18.109794 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.109460 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 08:49:18.109794 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.109643 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mjqd6\"" Jul 29 08:49:18.111213 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.110260 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 08:49:18.129732 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.129703 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-sn9p9"] Jul 29 08:49:18.147016 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.146935 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41e45e49-9c61-43f9-86a8-1a46d12a4544-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qks4q\" (UID: \"41e45e49-9c61-43f9-86a8-1a46d12a4544\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qks4q" Jul 29 08:49:18.147016 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.146978 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/41e45e49-9c61-43f9-86a8-1a46d12a4544-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-qks4q\" (UID: \"41e45e49-9c61-43f9-86a8-1a46d12a4544\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qks4q" Jul 29 08:49:18.147754 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.147706 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/41e45e49-9c61-43f9-86a8-1a46d12a4544-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-qks4q\" (UID: \"41e45e49-9c61-43f9-86a8-1a46d12a4544\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qks4q" Jul 29 08:49:18.149968 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.149941 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41e45e49-9c61-43f9-86a8-1a46d12a4544-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qks4q\" (UID: \"41e45e49-9c61-43f9-86a8-1a46d12a4544\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qks4q" Jul 29 08:49:18.203820 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.203788 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" Jul 29 08:49:18.247731 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.247695 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9drpv\" (UniqueName: \"kubernetes.io/projected/a96b2499-7e0c-4f91-8954-b33007b6753c-kube-api-access-9drpv\") pod \"downloads-7c97dc4555-sn9p9\" (UID: \"a96b2499-7e0c-4f91-8954-b33007b6753c\") " pod="openshift-console/downloads-7c97dc4555-sn9p9" Jul 29 08:49:18.299981 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.299943 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qks4q" Jul 29 08:49:18.348467 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.348432 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9drpv\" (UniqueName: \"kubernetes.io/projected/a96b2499-7e0c-4f91-8954-b33007b6753c-kube-api-access-9drpv\") pod \"downloads-7c97dc4555-sn9p9\" (UID: \"a96b2499-7e0c-4f91-8954-b33007b6753c\") " pod="openshift-console/downloads-7c97dc4555-sn9p9" Jul 29 08:49:18.379706 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.379680 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9drpv\" (UniqueName: \"kubernetes.io/projected/a96b2499-7e0c-4f91-8954-b33007b6753c-kube-api-access-9drpv\") pod \"downloads-7c97dc4555-sn9p9\" (UID: \"a96b2499-7e0c-4f91-8954-b33007b6753c\") " pod="openshift-console/downloads-7c97dc4555-sn9p9" Jul 29 08:49:18.420542 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:18.420510 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-sn9p9" Jul 29 08:49:20.405532 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.405495 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-qks4q"] Jul 29 08:49:20.408510 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:49:20.408481 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e45e49_9c61_43f9_86a8_1a46d12a4544.slice/crio-79210070cf0f85f602a9bd9406fbe28483b48448623cbace5b80ebce273d06b3 WatchSource:0}: Error finding container 79210070cf0f85f602a9bd9406fbe28483b48448623cbace5b80ebce273d06b3: Status 404 returned error can't find the container with id 79210070cf0f85f602a9bd9406fbe28483b48448623cbace5b80ebce273d06b3 Jul 29 08:49:20.434007 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.433098 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg"] Jul 29 08:49:20.437323 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.437258 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-sn9p9"] Jul 29 08:49:20.441354 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:49:20.441131 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f4aa74_7387_47a8_83bc_5bff5887fb2e.slice/crio-28c62a5ab4181497cae4f177d7799f7243c021182998e4317d7794c888bced24 WatchSource:0}: Error finding container 28c62a5ab4181497cae4f177d7799f7243c021182998e4317d7794c888bced24: Status 404 returned error can't find the container with id 28c62a5ab4181497cae4f177d7799f7243c021182998e4317d7794c888bced24 Jul 29 08:49:20.446956 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:49:20.446922 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96b2499_7e0c_4f91_8954_b33007b6753c.slice/crio-6ee057000b1885501f2a5168a6ff6544eb57d14b0e3e2ccb85a40e08d51ed540 WatchSource:0}: Error finding container 6ee057000b1885501f2a5168a6ff6544eb57d14b0e3e2ccb85a40e08d51ed540: Status 404 returned error can't find the container with id 6ee057000b1885501f2a5168a6ff6544eb57d14b0e3e2ccb85a40e08d51ed540 Jul 29 08:49:20.467899 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.467866 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7ff2z"] Jul 29 08:49:20.650638 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.650607 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nfxfj" event={"ID":"6c3c1c04-de29-4b84-bbb3-f6e5107979dd","Type":"ContainerStarted","Data":"6820800beed54e2cc2072196eb9777438a32e84a7adecc1e234db7862ee12487"} Jul 29 08:49:20.652448 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.652411 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9dgqn" event={"ID":"d5fac53f-30ee-496f-9131-c819ae8e4963","Type":"ContainerStarted","Data":"830cdfe32d8d913a057c05bcadede8604ef3605566366ac114527f523fb8b334"} Jul 29 08:49:20.652564 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.652457 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9dgqn" event={"ID":"d5fac53f-30ee-496f-9131-c819ae8e4963","Type":"ContainerStarted","Data":"f85502ab37f575e7bab4c7b4e2667d8af52cfd692938a4150f60e6c6e7f181c6"} Jul 29 08:49:20.653824 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.653799 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rwtwp" event={"ID":"3e7aa462-6508-4fa0-ab6d-43a5670db158","Type":"ContainerStarted","Data":"ccf9ec295ac528627378f4a5314d0c8fb027b231ce1e978d0d906a50fb1039ef"} Jul 29 08:49:20.653922 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.653859 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:49:20.654893 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.654875 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-sn9p9" event={"ID":"a96b2499-7e0c-4f91-8954-b33007b6753c","Type":"ContainerStarted","Data":"6ee057000b1885501f2a5168a6ff6544eb57d14b0e3e2ccb85a40e08d51ed540"} Jul 29 08:49:20.656200 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.656177 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z6kw7" event={"ID":"9014122b-1078-4e35-b53c-234e93bbd054","Type":"ContainerStarted","Data":"59327491fd7245ea2162a17f2d10c8f69438c3704dcec55b1f3bdb86d1e8bb3c"} Jul 29 08:49:20.657304 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.657256 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" event={"ID":"78f4aa74-7387-47a8-83bc-5bff5887fb2e","Type":"ContainerStarted","Data":"28c62a5ab4181497cae4f177d7799f7243c021182998e4317d7794c888bced24"} Jul 29 08:49:20.658312 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.658292 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qks4q" event={"ID":"41e45e49-9c61-43f9-86a8-1a46d12a4544","Type":"ContainerStarted","Data":"79210070cf0f85f602a9bd9406fbe28483b48448623cbace5b80ebce273d06b3"} Jul 29 08:49:20.659313 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.659295 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7ff2z" event={"ID":"bc60cc72-c068-4d53-a2ad-4141c76472a5","Type":"ContainerStarted","Data":"6d747d17e2a13377610734f671b0e822c0efdc926ddb582b911975a635be3e9d"} Jul 29 08:49:20.671037 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.670999 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9dgqn" podStartSLOduration=65.838670197 podStartE2EDuration="1m12.670986112s" totalImagesPullingTime="6.832315915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:20.670665835 +0000 UTC m=+73.794766638" watchObservedRunningTime="2026-07-29 08:49:20.670986112 +0000 UTC m=+73.795086908" Jul 29 08:49:20.689876 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.689812 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-z6kw7" podStartSLOduration=65.789987801 podStartE2EDuration="1m12.689794229s" totalImagesPullingTime="6.899806428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:20.689478862 +0000 UTC m=+73.813579663" watchObservedRunningTime="2026-07-29 08:49:20.689794229 +0000 UTC m=+73.813895028" Jul 29 08:49:20.712091 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:20.712001 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rwtwp" podStartSLOduration=66.135062113 podStartE2EDuration="1m12.711989151s" totalImagesPullingTime="6.576927038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:20.711496368 +0000 UTC m=+73.835597167" watchObservedRunningTime="2026-07-29 08:49:20.711989151 +0000 UTC m=+73.836089951" Jul 29 08:49:24.891901 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:24.891861 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68d6dff49c-ckvb5"] Jul 29 08:49:24.903699 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:24.903013 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:24.907153 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:24.906780 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 08:49:24.907153 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:24.907010 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 08:49:24.911558 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:24.911535 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 08:49:24.913055 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:24.911841 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 08:49:24.913055 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:24.912078 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 08:49:24.913055 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:24.912331 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fpl6s\"" Jul 29 08:49:24.917082 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:24.917009 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68d6dff49c-ckvb5"] Jul 29 08:49:25.018937 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.018611 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af6b391f-8894-4505-a5c2-185f56251278-console-serving-cert\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.018937 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.018677 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stf76\" (UniqueName: \"kubernetes.io/projected/af6b391f-8894-4505-a5c2-185f56251278-kube-api-access-stf76\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.018937 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.018709 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af6b391f-8894-4505-a5c2-185f56251278-console-oauth-config\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.018937 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.018749 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-console-config\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.018937 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.018786 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-oauth-serving-cert\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.018937 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.018815 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-service-ca\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.120391 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.119753 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af6b391f-8894-4505-a5c2-185f56251278-console-serving-cert\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.120391 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.119820 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stf76\" (UniqueName: \"kubernetes.io/projected/af6b391f-8894-4505-a5c2-185f56251278-kube-api-access-stf76\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.120391 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.119854 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af6b391f-8894-4505-a5c2-185f56251278-console-oauth-config\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.120391 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.119897 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-console-config\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.120391 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.119938 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-oauth-serving-cert\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.120391 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.119975 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-service-ca\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.120998 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.120946 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-oauth-serving-cert\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.121161 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.121120 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-console-config\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.121230 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.121190 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-service-ca\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.124142 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.123959 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af6b391f-8894-4505-a5c2-185f56251278-console-oauth-config\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.125002 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.124976 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af6b391f-8894-4505-a5c2-185f56251278-console-serving-cert\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.141065 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.141015 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stf76\" (UniqueName: \"kubernetes.io/projected/af6b391f-8894-4505-a5c2-185f56251278-kube-api-access-stf76\") pod \"console-68d6dff49c-ckvb5\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:25.222454 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:25.222393 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:26.611079 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:26.610187 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68d6dff49c-ckvb5"] Jul 29 08:49:26.692276 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:26.692238 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nfxfj" event={"ID":"6c3c1c04-de29-4b84-bbb3-f6e5107979dd","Type":"ContainerStarted","Data":"f7349dd7c7dd8064103ed5e550cfa75777abeffc71f73c240df4f57049f2bbd5"} Jul 29 08:49:26.694818 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:26.694781 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68d6dff49c-ckvb5" event={"ID":"af6b391f-8894-4505-a5c2-185f56251278","Type":"ContainerStarted","Data":"6caf80091aa8d5d634802e45ab9b1d4dfe3dbc7c34f9786b1625b457f3ec4f90"} Jul 29 08:49:26.697244 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:26.696826 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qks4q" event={"ID":"41e45e49-9c61-43f9-86a8-1a46d12a4544","Type":"ContainerStarted","Data":"7355e68b817db3ef858b297601e2be1d1cb4618ddbb8c49b60a173961ff630a8"} Jul 29 08:49:26.701120 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:26.700962 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7ff2z" event={"ID":"bc60cc72-c068-4d53-a2ad-4141c76472a5","Type":"ContainerStarted","Data":"abd3bedc717f764760421045d4947150b2c91aa4d92239cb3905b7ee742aa087"} Jul 29 08:49:26.701120 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:26.701060 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7ff2z" event={"ID":"bc60cc72-c068-4d53-a2ad-4141c76472a5","Type":"ContainerStarted","Data":"54023dde68d585614236cea21e325e57664306834f55bb4523bdb8c12d455473"} Jul 29 08:49:26.701120 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:26.701106 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7ff2z" Jul 29 08:49:26.715095 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:26.715015 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qks4q" podStartSLOduration=4.661070833 podStartE2EDuration="9.715000617s" totalImagesPullingTime="5.053929784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:26.714636439 +0000 UTC m=+79.838737240" watchObservedRunningTime="2026-07-29 08:49:26.715000617 +0000 UTC m=+79.839101453" Jul 29 08:49:26.732391 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:26.732098 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7ff2z" podStartSLOduration=39.747439204 podStartE2EDuration="44.732077058s" totalImagesPullingTime="4.984637854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:26.731141055 +0000 UTC m=+79.855241865" watchObservedRunningTime="2026-07-29 08:49:26.732077058 +0000 UTC m=+79.856177856" Jul 29 08:49:27.679708 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.679668 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b6d85bb57-6q42j"] Jul 29 08:49:27.683103 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.683077 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.699833 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.699806 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 08:49:27.718025 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.717964 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b6d85bb57-6q42j"] Jul 29 08:49:27.843618 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.843547 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-oauth-serving-cert\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.843618 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.843606 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-trusted-ca-bundle\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.843917 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.843740 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6780c80-d143-457f-a6c0-3d36df807598-console-serving-cert\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.843917 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.843780 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-service-ca\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.843917 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.843814 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-console-config\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.843917 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.843901 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6780c80-d143-457f-a6c0-3d36df807598-console-oauth-config\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.844184 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.844022 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-485dt\" (UniqueName: \"kubernetes.io/projected/e6780c80-d143-457f-a6c0-3d36df807598-kube-api-access-485dt\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.874001 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.873968 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8wcw7"] Jul 29 08:49:27.878184 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.878161 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8wcw7" Jul 29 08:49:27.881118 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.881088 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 29 08:49:27.881216 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.881116 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5hgjs\"" Jul 29 08:49:27.887712 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.886840 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8wcw7"] Jul 29 08:49:27.941345 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.941275 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:49:27.945511 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.945482 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-oauth-serving-cert\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.945680 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.945536 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-trusted-ca-bundle\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.945680 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.945568 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6780c80-d143-457f-a6c0-3d36df807598-console-serving-cert\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.945680 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.945595 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-service-ca\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.945680 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.945626 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-console-config\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.945680 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.945661 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6780c80-d143-457f-a6c0-3d36df807598-console-oauth-config\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.945936 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.945747 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-485dt\" (UniqueName: \"kubernetes.io/projected/e6780c80-d143-457f-a6c0-3d36df807598-kube-api-access-485dt\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.946316 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.946274 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-oauth-serving-cert\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.946530 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.946454 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-service-ca\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.946530 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.946454 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-console-config\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.947122 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.947099 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-trusted-ca-bundle\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.949202 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.949180 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6780c80-d143-457f-a6c0-3d36df807598-console-oauth-config\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.949957 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.949919 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6780c80-d143-457f-a6c0-3d36df807598-console-serving-cert\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.956192 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.956167 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-485dt\" (UniqueName: \"kubernetes.io/projected/e6780c80-d143-457f-a6c0-3d36df807598-kube-api-access-485dt\") pod \"console-7b6d85bb57-6q42j\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:27.994522 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:27.994486 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:28.046212 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:28.046176 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d88f07f1-d4c3-40fd-91d2-76ab5b41c546-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8wcw7\" (UID: \"d88f07f1-d4c3-40fd-91d2-76ab5b41c546\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8wcw7" Jul 29 08:49:28.147301 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:28.147259 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d88f07f1-d4c3-40fd-91d2-76ab5b41c546-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8wcw7\" (UID: \"d88f07f1-d4c3-40fd-91d2-76ab5b41c546\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8wcw7" Jul 29 08:49:28.150181 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:28.150152 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/d88f07f1-d4c3-40fd-91d2-76ab5b41c546-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-8wcw7\" (UID: \"d88f07f1-d4c3-40fd-91d2-76ab5b41c546\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8wcw7" Jul 29 08:49:28.189726 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:28.189578 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8wcw7" Jul 29 08:49:28.357488 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:28.357459 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8wcw7"] Jul 29 08:49:28.363660 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:49:28.363626 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88f07f1_d4c3_40fd_91d2_76ab5b41c546.slice/crio-ca4f75adf73ace91f2b4ceefe13ca61054dc9f7ec3479da43e7fd6833add8103 WatchSource:0}: Error finding container ca4f75adf73ace91f2b4ceefe13ca61054dc9f7ec3479da43e7fd6833add8103: Status 404 returned error can't find the container with id ca4f75adf73ace91f2b4ceefe13ca61054dc9f7ec3479da43e7fd6833add8103 Jul 29 08:49:28.380948 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:28.380919 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b6d85bb57-6q42j"] Jul 29 08:49:28.383932 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:49:28.383904 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6780c80_d143_457f_a6c0_3d36df807598.slice/crio-58f53f5924eb8c23f092b1e9974a314b439eab6c01630834e8bc670b957c28bb WatchSource:0}: Error finding container 58f53f5924eb8c23f092b1e9974a314b439eab6c01630834e8bc670b957c28bb: Status 404 returned error can't find the container with id 58f53f5924eb8c23f092b1e9974a314b439eab6c01630834e8bc670b957c28bb Jul 29 08:49:28.713139 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:28.711570 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" event={"ID":"78f4aa74-7387-47a8-83bc-5bff5887fb2e","Type":"ContainerStarted","Data":"177c98b4700a66bf4f1ca6119fdd79d8ef3b7a622f835525bf555210d9ce28c7"} Jul 29 08:49:28.713139 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:28.711989 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" Jul 29 08:49:28.714080 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:28.713998 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8wcw7" event={"ID":"d88f07f1-d4c3-40fd-91d2-76ab5b41c546","Type":"ContainerStarted","Data":"ca4f75adf73ace91f2b4ceefe13ca61054dc9f7ec3479da43e7fd6833add8103"} Jul 29 08:49:28.714194 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:28.714166 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" Jul 29 08:49:28.716452 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:28.715693 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6d85bb57-6q42j" event={"ID":"e6780c80-d143-457f-a6c0-3d36df807598","Type":"ContainerStarted","Data":"58f53f5924eb8c23f092b1e9974a314b439eab6c01630834e8bc670b957c28bb"} Jul 29 08:49:28.729133 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:28.728104 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d9b57869b-hlkpg" podStartSLOduration=3.881689712 podStartE2EDuration="11.728087428s" totalImagesPullingTime="7.846397716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:28.727882307 +0000 UTC m=+81.851983108" watchObservedRunningTime="2026-07-29 08:49:28.728087428 +0000 UTC m=+81.852188229" Jul 29 08:49:30.729973 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:30.729846 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8wcw7" event={"ID":"d88f07f1-d4c3-40fd-91d2-76ab5b41c546","Type":"ContainerStarted","Data":"2b6f2a7a1a4ced38204c8bb96f63eb508eb96d048bef7691e2d93b3e601660c0"} Jul 29 08:49:30.729973 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:30.729937 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8wcw7" Jul 29 08:49:30.736699 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:30.736637 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8wcw7" Jul 29 08:49:30.747514 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:30.747458 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-8wcw7" podStartSLOduration=1.765495112 podStartE2EDuration="3.747436787s" totalImagesPullingTime="1.981941675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:30.746464616 +0000 UTC m=+83.870565438" watchObservedRunningTime="2026-07-29 08:49:30.747436787 +0000 UTC m=+83.871537585" Jul 29 08:49:30.943169 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:30.943135 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-l5xff"] Jul 29 08:49:30.946757 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:30.946731 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" Jul 29 08:49:30.949651 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:30.949628 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 29 08:49:30.950809 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:30.950782 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 29 08:49:30.955669 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:30.951864 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 08:49:30.955669 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:30.952444 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 08:49:30.955669 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:30.953234 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-7mf4q\"" Jul 29 08:49:30.955669 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:30.953836 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 08:49:30.959840 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:30.959816 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-l5xff"] Jul 29 08:49:31.074056 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:31.073955 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f29ccd07-74c4-4c24-ab86-e1df11e9184b-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-l5xff\" (UID: \"f29ccd07-74c4-4c24-ab86-e1df11e9184b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" Jul 29 08:49:31.074056 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:31.074007 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw74t\" (UniqueName: \"kubernetes.io/projected/f29ccd07-74c4-4c24-ab86-e1df11e9184b-kube-api-access-qw74t\") pod \"prometheus-operator-84d69f794d-l5xff\" (UID: \"f29ccd07-74c4-4c24-ab86-e1df11e9184b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" Jul 29 08:49:31.074287 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:31.074069 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f29ccd07-74c4-4c24-ab86-e1df11e9184b-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-l5xff\" (UID: \"f29ccd07-74c4-4c24-ab86-e1df11e9184b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" Jul 29 08:49:31.074287 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:31.074118 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f29ccd07-74c4-4c24-ab86-e1df11e9184b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-l5xff\" (UID: \"f29ccd07-74c4-4c24-ab86-e1df11e9184b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" Jul 29 08:49:31.176156 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:31.175190 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f29ccd07-74c4-4c24-ab86-e1df11e9184b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-l5xff\" (UID: \"f29ccd07-74c4-4c24-ab86-e1df11e9184b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" Jul 29 08:49:31.176156 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:31.175257 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f29ccd07-74c4-4c24-ab86-e1df11e9184b-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-l5xff\" (UID: \"f29ccd07-74c4-4c24-ab86-e1df11e9184b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" Jul 29 08:49:31.176156 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:31.175286 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw74t\" (UniqueName: \"kubernetes.io/projected/f29ccd07-74c4-4c24-ab86-e1df11e9184b-kube-api-access-qw74t\") pod \"prometheus-operator-84d69f794d-l5xff\" (UID: \"f29ccd07-74c4-4c24-ab86-e1df11e9184b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" Jul 29 08:49:31.176156 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:31.175325 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f29ccd07-74c4-4c24-ab86-e1df11e9184b-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-l5xff\" (UID: \"f29ccd07-74c4-4c24-ab86-e1df11e9184b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" Jul 29 08:49:31.176156 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:31.176107 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f29ccd07-74c4-4c24-ab86-e1df11e9184b-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-l5xff\" (UID: \"f29ccd07-74c4-4c24-ab86-e1df11e9184b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" Jul 29 08:49:31.178542 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:31.178512 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f29ccd07-74c4-4c24-ab86-e1df11e9184b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-l5xff\" (UID: \"f29ccd07-74c4-4c24-ab86-e1df11e9184b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" Jul 29 08:49:31.178769 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:31.178722 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f29ccd07-74c4-4c24-ab86-e1df11e9184b-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-l5xff\" (UID: \"f29ccd07-74c4-4c24-ab86-e1df11e9184b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" Jul 29 08:49:31.186062 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:31.185996 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw74t\" (UniqueName: \"kubernetes.io/projected/f29ccd07-74c4-4c24-ab86-e1df11e9184b-kube-api-access-qw74t\") pod \"prometheus-operator-84d69f794d-l5xff\" (UID: \"f29ccd07-74c4-4c24-ab86-e1df11e9184b\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" Jul 29 08:49:31.260824 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:31.260788 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" Jul 29 08:49:36.707442 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:36.707411 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7ff2z" Jul 29 08:49:41.876495 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:41.876353 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-l5xff"] Jul 29 08:49:42.788508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:42.788461 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-sn9p9" event={"ID":"a96b2499-7e0c-4f91-8954-b33007b6753c","Type":"ContainerStarted","Data":"331428667f3ab36f291eeef52ee5e602c03bc29874c678493a29c65acc0a9f4d"} Jul 29 08:49:42.789692 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:42.789628 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-sn9p9" Jul 29 08:49:42.792516 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:42.792487 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68d6dff49c-ckvb5" event={"ID":"af6b391f-8894-4505-a5c2-185f56251278","Type":"ContainerStarted","Data":"5213d2097ee0160c5b21def6bfb10bdf22efd15f623410782d58ecd8114b542b"} Jul 29 08:49:42.793929 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:42.793891 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" event={"ID":"f29ccd07-74c4-4c24-ab86-e1df11e9184b","Type":"ContainerStarted","Data":"9da25265f2059f2845fcc934b12775f831c5a63e33cdf07b65581e30758fecb0"} Jul 29 08:49:42.797199 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:42.797135 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nfxfj" event={"ID":"6c3c1c04-de29-4b84-bbb3-f6e5107979dd","Type":"ContainerStarted","Data":"acbe132fb8d211cb11fb916990bd52d604ce8e73b26d4a44d5375a9ecbdee87c"} Jul 29 08:49:42.799807 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:42.799776 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6d85bb57-6q42j" event={"ID":"e6780c80-d143-457f-a6c0-3d36df807598","Type":"ContainerStarted","Data":"c686c2b04a06879913d7312cd3755582588213f2c0a8af0841a24ba3b45f1384"} Jul 29 08:49:42.801542 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:42.801521 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-sn9p9" Jul 29 08:49:42.854508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:42.854252 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-sn9p9" podStartSLOduration=3.523884928 podStartE2EDuration="24.85423027s" totalImagesPullingTime="21.330345342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:42.819349762 +0000 UTC m=+95.943450574" watchObservedRunningTime="2026-07-29 08:49:42.85423027 +0000 UTC m=+95.978331071" Jul 29 08:49:42.854508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:42.854377 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b6d85bb57-6q42j" podStartSLOduration=2.528145201 podStartE2EDuration="15.854356211s" totalImagesPullingTime="13.32621101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:42.850791026 +0000 UTC m=+95.974891824" watchObservedRunningTime="2026-07-29 08:49:42.854356211 +0000 UTC m=+95.978457012" Jul 29 08:49:42.915618 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:42.915440 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68d6dff49c-ckvb5" podStartSLOduration=3.822015711 podStartE2EDuration="18.915417638s" totalImagesPullingTime="15.093401927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:42.913598081 +0000 UTC m=+96.037698880" watchObservedRunningTime="2026-07-29 08:49:42.915417638 +0000 UTC m=+96.039518437" Jul 29 08:49:42.935627 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:42.935564 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nfxfj" podStartSLOduration=17.756119667 podStartE2EDuration="46.935542123s" totalImagesPullingTime="29.179422456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 08:48:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:42.933793541 +0000 UTC m=+96.057894340" watchObservedRunningTime="2026-07-29 08:49:42.935542123 +0000 UTC m=+96.059642924" Jul 29 08:49:42.955598 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:42.955540 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-798649959c-9mnkr" podUID="0b774629-0437-451c-9d63-777851e043e2" containerName="registry" containerID="cri-o://1d970c7156959030bc34fb2ad2e93f221e8ddea8d8efc4e43d349c9abc20fa28" gracePeriod=30 Jul 29 08:49:43.659603 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.659572 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:49:43.682454 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.682411 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0b774629-0437-451c-9d63-777851e043e2-image-registry-private-configuration\") pod \"0b774629-0437-451c-9d63-777851e043e2\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " Jul 29 08:49:43.682645 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.682464 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0b774629-0437-451c-9d63-777851e043e2-ca-trust-extracted\") pod \"0b774629-0437-451c-9d63-777851e043e2\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " Jul 29 08:49:43.682645 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.682509 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-bound-sa-token\") pod \"0b774629-0437-451c-9d63-777851e043e2\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " Jul 29 08:49:43.682645 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.682536 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b774629-0437-451c-9d63-777851e043e2-trusted-ca\") pod \"0b774629-0437-451c-9d63-777851e043e2\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " Jul 29 08:49:43.682645 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.682584 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls\") pod \"0b774629-0437-451c-9d63-777851e043e2\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " Jul 29 08:49:43.682645 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.682615 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0b774629-0437-451c-9d63-777851e043e2-registry-certificates\") pod \"0b774629-0437-451c-9d63-777851e043e2\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " Jul 29 08:49:43.682645 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.682644 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5tvn\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-kube-api-access-v5tvn\") pod \"0b774629-0437-451c-9d63-777851e043e2\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " Jul 29 08:49:43.682964 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.682716 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0b774629-0437-451c-9d63-777851e043e2-installation-pull-secrets\") pod \"0b774629-0437-451c-9d63-777851e043e2\" (UID: \"0b774629-0437-451c-9d63-777851e043e2\") " Jul 29 08:49:43.683996 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.683968 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b774629-0437-451c-9d63-777851e043e2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0b774629-0437-451c-9d63-777851e043e2" (UID: "0b774629-0437-451c-9d63-777851e043e2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:49:43.686827 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.686783 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b774629-0437-451c-9d63-777851e043e2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0b774629-0437-451c-9d63-777851e043e2" (UID: "0b774629-0437-451c-9d63-777851e043e2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:49:43.689694 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.689626 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b774629-0437-451c-9d63-777851e043e2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0b774629-0437-451c-9d63-777851e043e2" (UID: "0b774629-0437-451c-9d63-777851e043e2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:49:43.689694 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.689627 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-kube-api-access-v5tvn" (OuterVolumeSpecName: "kube-api-access-v5tvn") pod "0b774629-0437-451c-9d63-777851e043e2" (UID: "0b774629-0437-451c-9d63-777851e043e2"). InnerVolumeSpecName "kube-api-access-v5tvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:49:43.689939 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.689772 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0b774629-0437-451c-9d63-777851e043e2" (UID: "0b774629-0437-451c-9d63-777851e043e2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:49:43.690135 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.690097 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0b774629-0437-451c-9d63-777851e043e2" (UID: "0b774629-0437-451c-9d63-777851e043e2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:49:43.693583 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.692179 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b774629-0437-451c-9d63-777851e043e2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0b774629-0437-451c-9d63-777851e043e2" (UID: "0b774629-0437-451c-9d63-777851e043e2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:49:43.695873 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.695814 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b774629-0437-451c-9d63-777851e043e2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0b774629-0437-451c-9d63-777851e043e2" (UID: "0b774629-0437-451c-9d63-777851e043e2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:49:43.783418 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.783375 2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0b774629-0437-451c-9d63-777851e043e2-installation-pull-secrets\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:49:43.783418 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.783417 2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0b774629-0437-451c-9d63-777851e043e2-image-registry-private-configuration\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:49:43.783658 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.783435 2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0b774629-0437-451c-9d63-777851e043e2-ca-trust-extracted\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:49:43.783658 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.783450 2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-bound-sa-token\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:49:43.783658 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.783464 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b774629-0437-451c-9d63-777851e043e2-trusted-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:49:43.783658 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.783477 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-registry-tls\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:49:43.783658 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.783492 2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0b774629-0437-451c-9d63-777851e043e2-registry-certificates\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:49:43.783658 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.783505 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5tvn\" (UniqueName: \"kubernetes.io/projected/0b774629-0437-451c-9d63-777851e043e2-kube-api-access-v5tvn\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:49:43.805512 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.805476 2563 generic.go:356] "Generic (PLEG): container finished" podID="0b774629-0437-451c-9d63-777851e043e2" containerID="1d970c7156959030bc34fb2ad2e93f221e8ddea8d8efc4e43d349c9abc20fa28" exitCode=0 Jul 29 08:49:43.805699 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.805536 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-798649959c-9mnkr" Jul 29 08:49:43.805699 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.805566 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-798649959c-9mnkr" event={"ID":"0b774629-0437-451c-9d63-777851e043e2","Type":"ContainerDied","Data":"1d970c7156959030bc34fb2ad2e93f221e8ddea8d8efc4e43d349c9abc20fa28"} Jul 29 08:49:43.805699 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.805624 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-798649959c-9mnkr" event={"ID":"0b774629-0437-451c-9d63-777851e043e2","Type":"ContainerDied","Data":"fdcdcc0513bff85db0a081365764bafa7ef09b9f133a5dae1862826e4b770d91"} Jul 29 08:49:43.805699 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.805652 2563 scope.go:117] "RemoveContainer" containerID="1d970c7156959030bc34fb2ad2e93f221e8ddea8d8efc4e43d349c9abc20fa28" Jul 29 08:49:43.818552 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.818530 2563 scope.go:117] "RemoveContainer" containerID="1d970c7156959030bc34fb2ad2e93f221e8ddea8d8efc4e43d349c9abc20fa28" Jul 29 08:49:43.818865 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:49:43.818840 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d970c7156959030bc34fb2ad2e93f221e8ddea8d8efc4e43d349c9abc20fa28\": container with ID starting with 1d970c7156959030bc34fb2ad2e93f221e8ddea8d8efc4e43d349c9abc20fa28 not found: ID does not exist" containerID="1d970c7156959030bc34fb2ad2e93f221e8ddea8d8efc4e43d349c9abc20fa28" Jul 29 08:49:43.818962 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.818876 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d970c7156959030bc34fb2ad2e93f221e8ddea8d8efc4e43d349c9abc20fa28"} err="failed to get container status \"1d970c7156959030bc34fb2ad2e93f221e8ddea8d8efc4e43d349c9abc20fa28\": rpc error: code = NotFound desc = could not find container \"1d970c7156959030bc34fb2ad2e93f221e8ddea8d8efc4e43d349c9abc20fa28\": container with ID starting with 1d970c7156959030bc34fb2ad2e93f221e8ddea8d8efc4e43d349c9abc20fa28 not found: ID does not exist" Jul 29 08:49:43.831830 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.831799 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-798649959c-9mnkr"] Jul 29 08:49:43.837920 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:43.837897 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-798649959c-9mnkr"] Jul 29 08:49:44.813689 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:44.813626 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" event={"ID":"f29ccd07-74c4-4c24-ab86-e1df11e9184b","Type":"ContainerStarted","Data":"6f66222fbaade9da560df3c918823781afaebd6aeaed13ebe52efb7090262813"} Jul 29 08:49:44.814139 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:44.813740 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" event={"ID":"f29ccd07-74c4-4c24-ab86-e1df11e9184b","Type":"ContainerStarted","Data":"7e932e854b3b66cb34eaa7314cb6c9644c2fe8867c342789cc04d9c0b80ba115"} Jul 29 08:49:44.850566 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:44.850507 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-l5xff" podStartSLOduration=12.590895543 podStartE2EDuration="14.850485635s" totalImagesPullingTime="2.259590092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:44.850140026 +0000 UTC m=+97.974240826" watchObservedRunningTime="2026-07-29 08:49:44.850485635 +0000 UTC m=+97.974586436" Jul 29 08:49:45.222907 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:45.222861 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:45.223098 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:45.222946 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:45.228200 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:45.228175 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:45.363163 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:45.363123 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b774629-0437-451c-9d63-777851e043e2" path="/var/lib/kubelet/pods/0b774629-0437-451c-9d63-777851e043e2/volumes" Jul 29 08:49:45.823167 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:45.823137 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:49:46.467137 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.467105 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84"] Jul 29 08:49:46.467414 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.467398 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b774629-0437-451c-9d63-777851e043e2" containerName="registry" Jul 29 08:49:46.467414 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.467413 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b774629-0437-451c-9d63-777851e043e2" containerName="registry" Jul 29 08:49:46.467543 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.467485 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b774629-0437-451c-9d63-777851e043e2" containerName="registry" Jul 29 08:49:46.502925 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.502892 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jcvqx"] Jul 29 08:49:46.503102 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.503081 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.506733 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.506704 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 29 08:49:46.507398 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.507350 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 29 08:49:46.507691 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.507669 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 29 08:49:46.508629 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.508612 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gbz6l\"" Jul 29 08:49:46.523424 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.523353 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84"] Jul 29 08:49:46.523518 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.523466 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.527884 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.527864 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 08:49:46.528298 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.528275 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 08:49:46.528427 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.528300 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 08:49:46.528624 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.528607 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b8lr8\"" Jul 29 08:49:46.609630 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.609591 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ee3dae8-2df4-4b54-be97-532bebe98369-sys\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.609630 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.609626 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.609865 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.609653 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ee3dae8-2df4-4b54-be97-532bebe98369-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.609865 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.609714 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.609865 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.609791 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.609865 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.609819 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.609865 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.609861 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.610120 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.609888 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ee3dae8-2df4-4b54-be97-532bebe98369-node-exporter-tls\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.610120 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.609915 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ee3dae8-2df4-4b54-be97-532bebe98369-root\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.610120 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.609967 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ee3dae8-2df4-4b54-be97-532bebe98369-metrics-client-ca\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.610120 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.609999 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftrlg\" (UniqueName: \"kubernetes.io/projected/9ee3dae8-2df4-4b54-be97-532bebe98369-kube-api-access-ftrlg\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.610120 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.610030 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ee3dae8-2df4-4b54-be97-532bebe98369-node-exporter-textfile\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.610120 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.610049 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs24j\" (UniqueName: \"kubernetes.io/projected/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-kube-api-access-vs24j\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.610120 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.610064 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ee3dae8-2df4-4b54-be97-532bebe98369-node-exporter-wtmp\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.610120 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.610088 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ee3dae8-2df4-4b54-be97-532bebe98369-node-exporter-accelerators-collector-config\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.711410 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711355 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.711584 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711434 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.711584 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711464 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ee3dae8-2df4-4b54-be97-532bebe98369-node-exporter-tls\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.711584 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711485 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ee3dae8-2df4-4b54-be97-532bebe98369-root\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.711584 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711509 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ee3dae8-2df4-4b54-be97-532bebe98369-metrics-client-ca\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.711584 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711539 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrlg\" (UniqueName: \"kubernetes.io/projected/9ee3dae8-2df4-4b54-be97-532bebe98369-kube-api-access-ftrlg\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.711584 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711567 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ee3dae8-2df4-4b54-be97-532bebe98369-node-exporter-textfile\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.711884 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711601 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs24j\" (UniqueName: \"kubernetes.io/projected/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-kube-api-access-vs24j\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.711884 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711628 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ee3dae8-2df4-4b54-be97-532bebe98369-node-exporter-wtmp\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.711884 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711685 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ee3dae8-2df4-4b54-be97-532bebe98369-root\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.711884 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711738 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ee3dae8-2df4-4b54-be97-532bebe98369-node-exporter-accelerators-collector-config\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.711884 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711770 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ee3dae8-2df4-4b54-be97-532bebe98369-node-exporter-wtmp\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.711884 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711779 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ee3dae8-2df4-4b54-be97-532bebe98369-sys\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.711884 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711819 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ee3dae8-2df4-4b54-be97-532bebe98369-sys\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.711884 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711826 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.712269 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711903 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ee3dae8-2df4-4b54-be97-532bebe98369-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.712269 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.711953 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.712269 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.712013 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.712269 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.712166 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ee3dae8-2df4-4b54-be97-532bebe98369-metrics-client-ca\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.712504 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.712285 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ee3dae8-2df4-4b54-be97-532bebe98369-node-exporter-accelerators-collector-config\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.712588 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.712561 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ee3dae8-2df4-4b54-be97-532bebe98369-node-exporter-textfile\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.712646 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.712601 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.712903 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.712847 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.713070 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.713047 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.715119 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.714949 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.715119 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.715080 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ee3dae8-2df4-4b54-be97-532bebe98369-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.715274 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.715256 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ee3dae8-2df4-4b54-be97-532bebe98369-node-exporter-tls\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.715439 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.715420 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.726244 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.726189 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftrlg\" (UniqueName: \"kubernetes.io/projected/9ee3dae8-2df4-4b54-be97-532bebe98369-kube-api-access-ftrlg\") pod \"node-exporter-jcvqx\" (UID: \"9ee3dae8-2df4-4b54-be97-532bebe98369\") " pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.726584 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.726564 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs24j\" (UniqueName: \"kubernetes.io/projected/a44c7b97-b8d1-479c-868d-5b59ac55b6b7-kube-api-access-vs24j\") pod \"kube-state-metrics-56d9f944b4-9nc84\" (UID: \"a44c7b97-b8d1-479c-868d-5b59ac55b6b7\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.815217 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.815176 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" Jul 29 08:49:46.833493 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:46.833463 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jcvqx" Jul 29 08:49:46.885297 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:49:46.885257 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee3dae8_2df4_4b54_be97_532bebe98369.slice/crio-6febe14df5256d676847b3af03d6451a0d32f1239ac1ad2a622761e7418de6fb WatchSource:0}: Error finding container 6febe14df5256d676847b3af03d6451a0d32f1239ac1ad2a622761e7418de6fb: Status 404 returned error can't find the container with id 6febe14df5256d676847b3af03d6451a0d32f1239ac1ad2a622761e7418de6fb Jul 29 08:49:47.000165 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.000137 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84"] Jul 29 08:49:47.002978 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:49:47.002945 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44c7b97_b8d1_479c_868d_5b59ac55b6b7.slice/crio-55a6fc6c86510672edfb1a1e4d4ed4a38ffc003e13c714cfcfa6178f0206feec WatchSource:0}: Error finding container 55a6fc6c86510672edfb1a1e4d4ed4a38ffc003e13c714cfcfa6178f0206feec: Status 404 returned error can't find the container with id 55a6fc6c86510672edfb1a1e4d4ed4a38ffc003e13c714cfcfa6178f0206feec Jul 29 08:49:47.533752 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.533718 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 08:49:47.551315 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.551287 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.554903 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.554867 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 29 08:49:47.555245 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.555215 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 29 08:49:47.555650 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.555630 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 29 08:49:47.555855 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.555833 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ddd4r\"" Jul 29 08:49:47.555964 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.555926 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 29 08:49:47.556509 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.556489 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 29 08:49:47.558058 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.556947 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 29 08:49:47.558058 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.557933 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 29 08:49:47.558309 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.558232 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 29 08:49:47.562165 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.559504 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 08:49:47.562165 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.559793 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 29 08:49:47.623292 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.623163 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62fd3c7-480b-4c33-8a21-b9e566498f9b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.623292 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.623211 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a62fd3c7-480b-4c33-8a21-b9e566498f9b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.623292 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.623250 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a62fd3c7-480b-4c33-8a21-b9e566498f9b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.623292 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.623299 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a62fd3c7-480b-4c33-8a21-b9e566498f9b-config-out\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.623660 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.623342 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.623660 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.623416 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-web-config\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.623660 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.623481 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.623660 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.623512 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvxqj\" (UniqueName: \"kubernetes.io/projected/a62fd3c7-480b-4c33-8a21-b9e566498f9b-kube-api-access-fvxqj\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.623660 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.623553 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.623660 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.623617 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a62fd3c7-480b-4c33-8a21-b9e566498f9b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.623951 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.623673 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.623951 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.623704 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-config-volume\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.623951 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.623739 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.724146 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.724105 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62fd3c7-480b-4c33-8a21-b9e566498f9b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.724436 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.724161 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a62fd3c7-480b-4c33-8a21-b9e566498f9b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.724436 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.724201 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a62fd3c7-480b-4c33-8a21-b9e566498f9b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.724436 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.724233 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a62fd3c7-480b-4c33-8a21-b9e566498f9b-config-out\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.724436 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.724258 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.724436 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.724289 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-web-config\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.724436 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.724328 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.724436 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.724357 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvxqj\" (UniqueName: \"kubernetes.io/projected/a62fd3c7-480b-4c33-8a21-b9e566498f9b-kube-api-access-fvxqj\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.724436 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.724408 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.724436 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.724435 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a62fd3c7-480b-4c33-8a21-b9e566498f9b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.725077 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.724485 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.725077 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.724512 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-config-volume\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.725077 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.724543 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.725278 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:49:47.725258 2563 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 29 08:49:47.725356 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:49:47.725344 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-secret-alertmanager-main-tls podName:a62fd3c7-480b-4c33-8a21-b9e566498f9b nodeName:}" failed. No retries permitted until 2026-07-29 08:49:48.225320603 +0000 UTC m=+101.349421399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "a62fd3c7-480b-4c33-8a21-b9e566498f9b") : secret "alertmanager-main-tls" not found Jul 29 08:49:47.729964 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.729850 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62fd3c7-480b-4c33-8a21-b9e566498f9b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.730319 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.730238 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a62fd3c7-480b-4c33-8a21-b9e566498f9b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.731684 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.730764 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.731684 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.731451 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a62fd3c7-480b-4c33-8a21-b9e566498f9b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.731843 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.731696 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a62fd3c7-480b-4c33-8a21-b9e566498f9b-config-out\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.731903 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.731847 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.732613 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.732205 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-config-volume\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.732798 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.732748 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a62fd3c7-480b-4c33-8a21-b9e566498f9b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.732901 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.732857 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.733113 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.733086 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.734934 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.734895 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-web-config\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.740052 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.739979 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvxqj\" (UniqueName: \"kubernetes.io/projected/a62fd3c7-480b-4c33-8a21-b9e566498f9b-kube-api-access-fvxqj\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:47.833237 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.832659 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" event={"ID":"a44c7b97-b8d1-479c-868d-5b59ac55b6b7","Type":"ContainerStarted","Data":"55a6fc6c86510672edfb1a1e4d4ed4a38ffc003e13c714cfcfa6178f0206feec"} Jul 29 08:49:47.835221 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.835158 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jcvqx" event={"ID":"9ee3dae8-2df4-4b54-be97-532bebe98369","Type":"ContainerStarted","Data":"6febe14df5256d676847b3af03d6451a0d32f1239ac1ad2a622761e7418de6fb"} Jul 29 08:49:47.995761 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.995626 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:47.996808 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:47.996049 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:48.003533 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:48.003503 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:48.231170 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:48.231126 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:48.235600 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:48.235544 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a62fd3c7-480b-4c33-8a21-b9e566498f9b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a62fd3c7-480b-4c33-8a21-b9e566498f9b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:48.469176 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:48.469137 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 08:49:48.844217 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:48.844148 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:49:48.923508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:48.923475 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68d6dff49c-ckvb5"] Jul 29 08:49:49.207588 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:49.207560 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 08:49:49.209972 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:49:49.209937 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62fd3c7_480b_4c33_8a21_b9e566498f9b.slice/crio-59b3947e289d74d93038bbf88d8b991b43a7ec49c7c6ff64bd14c65f38671b2c WatchSource:0}: Error finding container 59b3947e289d74d93038bbf88d8b991b43a7ec49c7c6ff64bd14c65f38671b2c: Status 404 returned error can't find the container with id 59b3947e289d74d93038bbf88d8b991b43a7ec49c7c6ff64bd14c65f38671b2c Jul 29 08:49:49.846465 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:49.846420 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a62fd3c7-480b-4c33-8a21-b9e566498f9b","Type":"ContainerStarted","Data":"59b3947e289d74d93038bbf88d8b991b43a7ec49c7c6ff64bd14c65f38671b2c"} Jul 29 08:49:49.849495 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:49.849459 2563 generic.go:356] "Generic (PLEG): container finished" podID="9ee3dae8-2df4-4b54-be97-532bebe98369" containerID="df0174ea091e31253339fd957411b957116df8cc6a7ace275a4d6eee387b3a9d" exitCode=0 Jul 29 08:49:49.849728 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:49.849645 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jcvqx" event={"ID":"9ee3dae8-2df4-4b54-be97-532bebe98369","Type":"ContainerDied","Data":"df0174ea091e31253339fd957411b957116df8cc6a7ace275a4d6eee387b3a9d"} Jul 29 08:49:49.854392 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:49.854047 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" event={"ID":"a44c7b97-b8d1-479c-868d-5b59ac55b6b7","Type":"ContainerStarted","Data":"8fbee3c8c3aebe8f113f0dd48a559ed7da764098a3e5c4ee120df7bd549aa979"} Jul 29 08:49:49.854392 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:49.854103 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" event={"ID":"a44c7b97-b8d1-479c-868d-5b59ac55b6b7","Type":"ContainerStarted","Data":"a7bdd85408578f0f8c00c3eb7910817acf7a1184eb0d90f0dee3d45cadf817f9"} Jul 29 08:49:49.854392 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:49.854132 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" event={"ID":"a44c7b97-b8d1-479c-868d-5b59ac55b6b7","Type":"ContainerStarted","Data":"84e60c63788734f69e0dc810b9ac86785ec7f87bc648248bb8b844970db78b1d"} Jul 29 08:49:49.930350 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:49.928401 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9nc84" podStartSLOduration=1.900636202 podStartE2EDuration="3.92837803s" totalImagesPullingTime="2.027741828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:49.925931823 +0000 UTC m=+103.050032622" watchObservedRunningTime="2026-07-29 08:49:49.92837803 +0000 UTC m=+103.052478829" Jul 29 08:49:50.852663 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.852618 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg"] Jul 29 08:49:50.874310 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.874273 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jcvqx" event={"ID":"9ee3dae8-2df4-4b54-be97-532bebe98369","Type":"ContainerStarted","Data":"e6ada58677387b0ea2586ae4ea751458bf65a05d29c0b4dfba073a524e1286bc"} Jul 29 08:49:50.874510 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.874328 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg"] Jul 29 08:49:50.874510 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.874353 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jcvqx" event={"ID":"9ee3dae8-2df4-4b54-be97-532bebe98369","Type":"ContainerStarted","Data":"3db333673af8896f0a762a938d7fca9893b036a46a8fad35145ea1b1f70ecc6e"} Jul 29 08:49:50.874510 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.874432 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:50.877488 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.877463 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bvnsx\"" Jul 29 08:49:50.877624 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.877463 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 29 08:49:50.877624 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.877613 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ctp41fnh64vr\"" Jul 29 08:49:50.877732 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.877708 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 29 08:49:50.877855 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.877838 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 29 08:49:50.877930 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.877913 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 29 08:49:50.886217 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.886153 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jcvqx" podStartSLOduration=2.638671434 podStartE2EDuration="4.886138665s" totalImagesPullingTime="2.142120735s" totalInitContainerRuntime="105.346496ms" isStatefulPod=true podCreationTimestamp="2026-07-29 08:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:50.883918184 +0000 UTC m=+104.008018995" watchObservedRunningTime="2026-07-29 08:49:50.886138665 +0000 UTC m=+104.010239491" Jul 29 08:49:50.960231 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.960197 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ecaff838-b293-4dfb-96b0-aeb81daf695b-secret-metrics-server-tls\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:50.960393 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.960264 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvjln\" (UniqueName: \"kubernetes.io/projected/ecaff838-b293-4dfb-96b0-aeb81daf695b-kube-api-access-bvjln\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:50.960393 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.960306 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ecaff838-b293-4dfb-96b0-aeb81daf695b-client-ca-bundle\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:50.960393 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.960352 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecaff838-b293-4dfb-96b0-aeb81daf695b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:50.960561 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.960437 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ecaff838-b293-4dfb-96b0-aeb81daf695b-secret-metrics-server-client-certs\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:50.960561 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.960470 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ecaff838-b293-4dfb-96b0-aeb81daf695b-audit-log\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:50.960561 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:50.960494 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ecaff838-b293-4dfb-96b0-aeb81daf695b-metrics-server-audit-profiles\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.062274 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.062239 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ecaff838-b293-4dfb-96b0-aeb81daf695b-secret-metrics-server-tls\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.062442 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.062301 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjln\" (UniqueName: \"kubernetes.io/projected/ecaff838-b293-4dfb-96b0-aeb81daf695b-kube-api-access-bvjln\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.062442 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.062336 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ecaff838-b293-4dfb-96b0-aeb81daf695b-client-ca-bundle\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.062442 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.062397 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecaff838-b293-4dfb-96b0-aeb81daf695b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.062442 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.062440 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ecaff838-b293-4dfb-96b0-aeb81daf695b-secret-metrics-server-client-certs\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.062666 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.062465 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ecaff838-b293-4dfb-96b0-aeb81daf695b-audit-log\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.062666 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.062491 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ecaff838-b293-4dfb-96b0-aeb81daf695b-metrics-server-audit-profiles\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.063731 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.063677 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ecaff838-b293-4dfb-96b0-aeb81daf695b-metrics-server-audit-profiles\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.063854 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.063827 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ecaff838-b293-4dfb-96b0-aeb81daf695b-audit-log\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.064443 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.064415 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecaff838-b293-4dfb-96b0-aeb81daf695b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.066520 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.066472 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ecaff838-b293-4dfb-96b0-aeb81daf695b-secret-metrics-server-tls\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.066609 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.066584 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ecaff838-b293-4dfb-96b0-aeb81daf695b-secret-metrics-server-client-certs\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.067079 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.067026 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ecaff838-b293-4dfb-96b0-aeb81daf695b-client-ca-bundle\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.075657 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.075637 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvjln\" (UniqueName: \"kubernetes.io/projected/ecaff838-b293-4dfb-96b0-aeb81daf695b-kube-api-access-bvjln\") pod \"metrics-server-9b4dd9b9c-cxdrg\" (UID: \"ecaff838-b293-4dfb-96b0-aeb81daf695b\") " pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.102783 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.102708 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx"] Jul 29 08:49:51.122801 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.122766 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx"] Jul 29 08:49:51.122923 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.122876 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx" Jul 29 08:49:51.125979 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.125955 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-r57zw\"" Jul 29 08:49:51.126100 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.126012 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 29 08:49:51.163156 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.163122 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjncs\" (UniqueName: \"kubernetes.io/projected/baa0986b-a86d-4acf-a9a7-0590bf571e1b-kube-api-access-rjncs\") pod \"monitoring-plugin-68b7fc9574-z4dvx\" (UID: \"baa0986b-a86d-4acf-a9a7-0590bf571e1b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx" Jul 29 08:49:51.163305 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.163164 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/baa0986b-a86d-4acf-a9a7-0590bf571e1b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-z4dvx\" (UID: \"baa0986b-a86d-4acf-a9a7-0590bf571e1b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx" Jul 29 08:49:51.186050 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.186015 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:49:51.264793 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.264565 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjncs\" (UniqueName: \"kubernetes.io/projected/baa0986b-a86d-4acf-a9a7-0590bf571e1b-kube-api-access-rjncs\") pod \"monitoring-plugin-68b7fc9574-z4dvx\" (UID: \"baa0986b-a86d-4acf-a9a7-0590bf571e1b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx" Jul 29 08:49:51.264793 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.264635 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/baa0986b-a86d-4acf-a9a7-0590bf571e1b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-z4dvx\" (UID: \"baa0986b-a86d-4acf-a9a7-0590bf571e1b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx" Jul 29 08:49:51.268228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.268190 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/baa0986b-a86d-4acf-a9a7-0590bf571e1b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-z4dvx\" (UID: \"baa0986b-a86d-4acf-a9a7-0590bf571e1b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx" Jul 29 08:49:51.274539 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.274514 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjncs\" (UniqueName: \"kubernetes.io/projected/baa0986b-a86d-4acf-a9a7-0590bf571e1b-kube-api-access-rjncs\") pod \"monitoring-plugin-68b7fc9574-z4dvx\" (UID: \"baa0986b-a86d-4acf-a9a7-0590bf571e1b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx" Jul 29 08:49:51.345095 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.345070 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg"] Jul 29 08:49:51.347541 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:49:51.347513 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecaff838_b293_4dfb_96b0_aeb81daf695b.slice/crio-ab6dc911b4faa2b9d00527839aa7c7ed464741c1e724eafec54d67b42877ed8e WatchSource:0}: Error finding container ab6dc911b4faa2b9d00527839aa7c7ed464741c1e724eafec54d67b42877ed8e: Status 404 returned error can't find the container with id ab6dc911b4faa2b9d00527839aa7c7ed464741c1e724eafec54d67b42877ed8e Jul 29 08:49:51.472617 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.472580 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx" Jul 29 08:49:51.535166 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.535122 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f8bf6b87c-69xgb"] Jul 29 08:49:51.560117 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.560089 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f8bf6b87c-69xgb"] Jul 29 08:49:51.560272 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.560183 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.641542 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.641375 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx"] Jul 29 08:49:51.644689 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:49:51.644657 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa0986b_a86d_4acf_a9a7_0590bf571e1b.slice/crio-b66e911f0693621741ad5cb8518b89c605e36d70edb40ca9243b1ed33c80f8e5 WatchSource:0}: Error finding container b66e911f0693621741ad5cb8518b89c605e36d70edb40ca9243b1ed33c80f8e5: Status 404 returned error can't find the container with id b66e911f0693621741ad5cb8518b89c605e36d70edb40ca9243b1ed33c80f8e5 Jul 29 08:49:51.667613 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.667590 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rwtwp" Jul 29 08:49:51.667834 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.667812 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-serving-cert\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.667883 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.667850 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-oauth-serving-cert\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.667883 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.667869 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s6qh\" (UniqueName: \"kubernetes.io/projected/92f0a2b3-1573-415b-9672-a9a1b51afdda-kube-api-access-7s6qh\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.667977 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.667902 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-config\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.667977 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.667917 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-trusted-ca-bundle\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.667977 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.667935 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-service-ca\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.668111 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.668069 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-oauth-config\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.769552 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.769471 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-oauth-config\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.769552 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.769545 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-serving-cert\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.769748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.769572 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-oauth-serving-cert\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.769748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.769588 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s6qh\" (UniqueName: \"kubernetes.io/projected/92f0a2b3-1573-415b-9672-a9a1b51afdda-kube-api-access-7s6qh\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.769748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.769619 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-config\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.769748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.769663 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-trusted-ca-bundle\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.769748 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.769693 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-service-ca\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.770523 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.770496 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-oauth-serving-cert\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.770655 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.770526 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-config\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.770655 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.770597 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-service-ca\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.770777 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.770666 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-trusted-ca-bundle\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.772042 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.772014 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-serving-cert\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.772042 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.772024 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-oauth-config\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.792626 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.792604 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s6qh\" (UniqueName: \"kubernetes.io/projected/92f0a2b3-1573-415b-9672-a9a1b51afdda-kube-api-access-7s6qh\") pod \"console-5f8bf6b87c-69xgb\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:51.866993 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.866958 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" event={"ID":"ecaff838-b293-4dfb-96b0-aeb81daf695b","Type":"ContainerStarted","Data":"ab6dc911b4faa2b9d00527839aa7c7ed464741c1e724eafec54d67b42877ed8e"} Jul 29 08:49:51.868165 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.868135 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx" event={"ID":"baa0986b-a86d-4acf-a9a7-0590bf571e1b","Type":"ContainerStarted","Data":"b66e911f0693621741ad5cb8518b89c605e36d70edb40ca9243b1ed33c80f8e5"} Jul 29 08:49:51.869611 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.869585 2563 generic.go:356] "Generic (PLEG): container finished" podID="a62fd3c7-480b-4c33-8a21-b9e566498f9b" containerID="246d1289a547040c9863621f69f89f8315be7bc0fdd0f0bff44aa026e42f4cd2" exitCode=0 Jul 29 08:49:51.869696 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.869674 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a62fd3c7-480b-4c33-8a21-b9e566498f9b","Type":"ContainerDied","Data":"246d1289a547040c9863621f69f89f8315be7bc0fdd0f0bff44aa026e42f4cd2"} Jul 29 08:49:51.872337 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:51.872319 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:49:52.051743 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:52.051646 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f8bf6b87c-69xgb"] Jul 29 08:49:52.056505 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:49:52.056478 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f0a2b3_1573_415b_9672_a9a1b51afdda.slice/crio-b5eff873317300ef2f4a1219b4c486c1931a451e42b02ba2c0e88348df979992 WatchSource:0}: Error finding container b5eff873317300ef2f4a1219b4c486c1931a451e42b02ba2c0e88348df979992: Status 404 returned error can't find the container with id b5eff873317300ef2f4a1219b4c486c1931a451e42b02ba2c0e88348df979992 Jul 29 08:49:52.876954 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:52.876911 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8bf6b87c-69xgb" event={"ID":"92f0a2b3-1573-415b-9672-a9a1b51afdda","Type":"ContainerStarted","Data":"bbe2d02bfff0e7bb71511923460c3252db1126eb4779c5d4ccdd913cc6c8e2fe"} Jul 29 08:49:52.877458 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:52.877017 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8bf6b87c-69xgb" event={"ID":"92f0a2b3-1573-415b-9672-a9a1b51afdda","Type":"ContainerStarted","Data":"b5eff873317300ef2f4a1219b4c486c1931a451e42b02ba2c0e88348df979992"} Jul 29 08:49:52.942846 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:52.942786 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f8bf6b87c-69xgb" podStartSLOduration=1.942767769 podStartE2EDuration="1.942767769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:52.942314072 +0000 UTC m=+106.066414871" watchObservedRunningTime="2026-07-29 08:49:52.942767769 +0000 UTC m=+106.066868567" Jul 29 08:49:55.893386 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:55.893278 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a62fd3c7-480b-4c33-8a21-b9e566498f9b","Type":"ContainerStarted","Data":"fee2ff426e5bc542498631b69f26e37fd9316aec71d9e5c16dbc7d9ccf31d246"} Jul 29 08:49:55.893805 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:55.893405 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a62fd3c7-480b-4c33-8a21-b9e566498f9b","Type":"ContainerStarted","Data":"df123b2d498822b5458e051d2027e35e23410f1cf75a9b44b07a5425e3086f30"} Jul 29 08:49:55.893805 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:55.893448 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a62fd3c7-480b-4c33-8a21-b9e566498f9b","Type":"ContainerStarted","Data":"3de62bd7ed2fd2130311ceab05c30a8c8ec595ac531c3592feacf24cd4d90534"} Jul 29 08:49:55.893805 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:55.893488 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a62fd3c7-480b-4c33-8a21-b9e566498f9b","Type":"ContainerStarted","Data":"8b71ffca487ffa0436cc7d0869a1f5f8358a81ac8c63087fd7427c81ea1b726c"} Jul 29 08:49:55.893805 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:55.893531 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a62fd3c7-480b-4c33-8a21-b9e566498f9b","Type":"ContainerStarted","Data":"90ee152d943ee84410c98c9e9791815516796c1b40d2715abe3012fafa9ed6ca"} Jul 29 08:49:55.894735 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:55.894711 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" event={"ID":"ecaff838-b293-4dfb-96b0-aeb81daf695b","Type":"ContainerStarted","Data":"64dab0747c3a610ac78e7aab73018fafc5adfa9cdc8ccf0598e4ab28e5958971"} Jul 29 08:49:55.896140 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:55.896115 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx" event={"ID":"baa0986b-a86d-4acf-a9a7-0590bf571e1b","Type":"ContainerStarted","Data":"349845d6fd25bb51f7a945e7c3187e99a9004860b393e04dc6307d9b2b400921"} Jul 29 08:49:55.896329 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:55.896310 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx" Jul 29 08:49:55.901077 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:55.901055 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx" Jul 29 08:49:55.918505 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:55.918461 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" podStartSLOduration=2.276881842 podStartE2EDuration="5.918447401s" totalImagesPullingTime="3.641565559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:55.917187181 +0000 UTC m=+109.041288000" watchObservedRunningTime="2026-07-29 08:49:55.918447401 +0000 UTC m=+109.042548200" Jul 29 08:49:55.939442 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:55.939393 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-z4dvx" podStartSLOduration=1.5942785590000002 podStartE2EDuration="4.939376488s" totalImagesPullingTime="3.345097929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:55.939068428 +0000 UTC m=+109.063169247" watchObservedRunningTime="2026-07-29 08:49:55.939376488 +0000 UTC m=+109.063477278" Jul 29 08:49:57.908593 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:57.908551 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a62fd3c7-480b-4c33-8a21-b9e566498f9b","Type":"ContainerStarted","Data":"980778421f7b996af0587829c3ef5943e07f6da2e7fcd2100d81564e99bd3f36"} Jul 29 08:49:57.939177 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:57.939118 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.747093431 podStartE2EDuration="10.939095776s" totalImagesPullingTime="6.166202914s" totalInitContainerRuntime="25.799431ms" isStatefulPod=false podCreationTimestamp="2026-07-29 08:49:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:49:57.937228494 +0000 UTC m=+111.061329318" watchObservedRunningTime="2026-07-29 08:49:57.939095776 +0000 UTC m=+111.063196576" Jul 29 08:49:58.188423 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:49:58.188390 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f8bf6b87c-69xgb"] Jul 29 08:50:01.873254 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:01.873212 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:50:08.954222 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:08.954141 2563 generic.go:356] "Generic (PLEG): container finished" podID="1e723fb2-089d-411b-9a4b-07d8d1b4cfa1" containerID="e39f415371a29812ac31c6212bda2074da2c10cdebd22df36c6381f106f560bc" exitCode=0 Jul 29 08:50:08.954222 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:08.954213 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" event={"ID":"1e723fb2-089d-411b-9a4b-07d8d1b4cfa1","Type":"ContainerDied","Data":"e39f415371a29812ac31c6212bda2074da2c10cdebd22df36c6381f106f560bc"} Jul 29 08:50:08.954644 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:08.954597 2563 scope.go:117] "RemoveContainer" containerID="e39f415371a29812ac31c6212bda2074da2c10cdebd22df36c6381f106f560bc" Jul 29 08:50:09.440772 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:09.440737 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b6d85bb57-6q42j"] Jul 29 08:50:09.959824 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:09.959793 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-8s2j9" event={"ID":"1e723fb2-089d-411b-9a4b-07d8d1b4cfa1","Type":"ContainerStarted","Data":"b3088d781e10dcdc6175ae4eda271c3426db1a5e6295ba42734666a1a1e3dca0"} Jul 29 08:50:11.186740 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:11.186704 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:50:11.187133 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:11.186784 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:50:13.947753 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:13.947707 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68d6dff49c-ckvb5" podUID="af6b391f-8894-4505-a5c2-185f56251278" containerName="console" containerID="cri-o://5213d2097ee0160c5b21def6bfb10bdf22efd15f623410782d58ecd8114b542b" gracePeriod=15 Jul 29 08:50:14.192011 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.191986 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:50:14.263674 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.263590 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-console-config\") pod \"af6b391f-8894-4505-a5c2-185f56251278\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " Jul 29 08:50:14.263674 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.263644 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stf76\" (UniqueName: \"kubernetes.io/projected/af6b391f-8894-4505-a5c2-185f56251278-kube-api-access-stf76\") pod \"af6b391f-8894-4505-a5c2-185f56251278\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " Jul 29 08:50:14.263896 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.263681 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af6b391f-8894-4505-a5c2-185f56251278-console-oauth-config\") pod \"af6b391f-8894-4505-a5c2-185f56251278\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " Jul 29 08:50:14.263896 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.263718 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af6b391f-8894-4505-a5c2-185f56251278-console-serving-cert\") pod \"af6b391f-8894-4505-a5c2-185f56251278\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " Jul 29 08:50:14.263896 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.263779 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-service-ca\") pod \"af6b391f-8894-4505-a5c2-185f56251278\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " Jul 29 08:50:14.263896 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.263806 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-oauth-serving-cert\") pod \"af6b391f-8894-4505-a5c2-185f56251278\" (UID: \"af6b391f-8894-4505-a5c2-185f56251278\") " Jul 29 08:50:14.264194 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.264116 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-console-config" (OuterVolumeSpecName: "console-config") pod "af6b391f-8894-4505-a5c2-185f56251278" (UID: "af6b391f-8894-4505-a5c2-185f56251278"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:50:14.264328 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.264231 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "af6b391f-8894-4505-a5c2-185f56251278" (UID: "af6b391f-8894-4505-a5c2-185f56251278"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:50:14.264328 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.264230 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-service-ca" (OuterVolumeSpecName: "service-ca") pod "af6b391f-8894-4505-a5c2-185f56251278" (UID: "af6b391f-8894-4505-a5c2-185f56251278"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:50:14.265952 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.265926 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af6b391f-8894-4505-a5c2-185f56251278-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "af6b391f-8894-4505-a5c2-185f56251278" (UID: "af6b391f-8894-4505-a5c2-185f56251278"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:50:14.266060 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.265960 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af6b391f-8894-4505-a5c2-185f56251278-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "af6b391f-8894-4505-a5c2-185f56251278" (UID: "af6b391f-8894-4505-a5c2-185f56251278"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:50:14.266060 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.266040 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6b391f-8894-4505-a5c2-185f56251278-kube-api-access-stf76" (OuterVolumeSpecName: "kube-api-access-stf76") pod "af6b391f-8894-4505-a5c2-185f56251278" (UID: "af6b391f-8894-4505-a5c2-185f56251278"). InnerVolumeSpecName "kube-api-access-stf76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:50:14.364871 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.364828 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-console-config\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:14.364871 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.364867 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stf76\" (UniqueName: \"kubernetes.io/projected/af6b391f-8894-4505-a5c2-185f56251278-kube-api-access-stf76\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:14.364871 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.364878 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af6b391f-8894-4505-a5c2-185f56251278-console-oauth-config\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:14.364871 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.364887 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af6b391f-8894-4505-a5c2-185f56251278-console-serving-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:14.365102 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.364896 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:14.365102 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.364904 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af6b391f-8894-4505-a5c2-185f56251278-oauth-serving-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:14.984548 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.984514 2563 generic.go:356] "Generic (PLEG): container finished" podID="af6b391f-8894-4505-a5c2-185f56251278" containerID="5213d2097ee0160c5b21def6bfb10bdf22efd15f623410782d58ecd8114b542b" exitCode=0 Jul 29 08:50:14.984982 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.984556 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68d6dff49c-ckvb5" event={"ID":"af6b391f-8894-4505-a5c2-185f56251278","Type":"ContainerDied","Data":"5213d2097ee0160c5b21def6bfb10bdf22efd15f623410782d58ecd8114b542b"} Jul 29 08:50:14.984982 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.984561 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68d6dff49c-ckvb5" Jul 29 08:50:14.984982 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.984599 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68d6dff49c-ckvb5" event={"ID":"af6b391f-8894-4505-a5c2-185f56251278","Type":"ContainerDied","Data":"6caf80091aa8d5d634802e45ab9b1d4dfe3dbc7c34f9786b1625b457f3ec4f90"} Jul 29 08:50:14.984982 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.984622 2563 scope.go:117] "RemoveContainer" containerID="5213d2097ee0160c5b21def6bfb10bdf22efd15f623410782d58ecd8114b542b" Jul 29 08:50:14.992966 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.992949 2563 scope.go:117] "RemoveContainer" containerID="5213d2097ee0160c5b21def6bfb10bdf22efd15f623410782d58ecd8114b542b" Jul 29 08:50:14.993208 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:50:14.993190 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5213d2097ee0160c5b21def6bfb10bdf22efd15f623410782d58ecd8114b542b\": container with ID starting with 5213d2097ee0160c5b21def6bfb10bdf22efd15f623410782d58ecd8114b542b not found: ID does not exist" containerID="5213d2097ee0160c5b21def6bfb10bdf22efd15f623410782d58ecd8114b542b" Jul 29 08:50:14.993252 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:14.993218 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5213d2097ee0160c5b21def6bfb10bdf22efd15f623410782d58ecd8114b542b"} err="failed to get container status \"5213d2097ee0160c5b21def6bfb10bdf22efd15f623410782d58ecd8114b542b\": rpc error: code = NotFound desc = could not find container \"5213d2097ee0160c5b21def6bfb10bdf22efd15f623410782d58ecd8114b542b\": container with ID starting with 5213d2097ee0160c5b21def6bfb10bdf22efd15f623410782d58ecd8114b542b not found: ID does not exist" Jul 29 08:50:15.007019 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:15.006986 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68d6dff49c-ckvb5"] Jul 29 08:50:15.011155 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:15.011131 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68d6dff49c-ckvb5"] Jul 29 08:50:15.361445 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:15.361346 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af6b391f-8894-4505-a5c2-185f56251278" path="/var/lib/kubelet/pods/af6b391f-8894-4505-a5c2-185f56251278/volumes" Jul 29 08:50:23.209179 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.209137 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f8bf6b87c-69xgb" podUID="92f0a2b3-1573-415b-9672-a9a1b51afdda" containerName="console" containerID="cri-o://bbe2d02bfff0e7bb71511923460c3252db1126eb4779c5d4ccdd913cc6c8e2fe" gracePeriod=15 Jul 29 08:50:23.478586 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.478507 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:50:23.641096 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.641057 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-serving-cert\") pod \"92f0a2b3-1573-415b-9672-a9a1b51afdda\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " Jul 29 08:50:23.641296 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.641155 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-service-ca\") pod \"92f0a2b3-1573-415b-9672-a9a1b51afdda\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " Jul 29 08:50:23.641389 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.641317 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-config\") pod \"92f0a2b3-1573-415b-9672-a9a1b51afdda\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " Jul 29 08:50:23.641389 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.641385 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-trusted-ca-bundle\") pod \"92f0a2b3-1573-415b-9672-a9a1b51afdda\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " Jul 29 08:50:23.641508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.641404 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-oauth-config\") pod \"92f0a2b3-1573-415b-9672-a9a1b51afdda\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " Jul 29 08:50:23.641508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.641450 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s6qh\" (UniqueName: \"kubernetes.io/projected/92f0a2b3-1573-415b-9672-a9a1b51afdda-kube-api-access-7s6qh\") pod \"92f0a2b3-1573-415b-9672-a9a1b51afdda\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " Jul 29 08:50:23.641508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.641487 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-oauth-serving-cert\") pod \"92f0a2b3-1573-415b-9672-a9a1b51afdda\" (UID: \"92f0a2b3-1573-415b-9672-a9a1b51afdda\") " Jul 29 08:50:23.641666 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.641641 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-service-ca" (OuterVolumeSpecName: "service-ca") pod "92f0a2b3-1573-415b-9672-a9a1b51afdda" (UID: "92f0a2b3-1573-415b-9672-a9a1b51afdda"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:50:23.641799 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.641777 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:23.641904 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.641765 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "92f0a2b3-1573-415b-9672-a9a1b51afdda" (UID: "92f0a2b3-1573-415b-9672-a9a1b51afdda"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:50:23.641904 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.641796 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-config" (OuterVolumeSpecName: "console-config") pod "92f0a2b3-1573-415b-9672-a9a1b51afdda" (UID: "92f0a2b3-1573-415b-9672-a9a1b51afdda"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:50:23.641904 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.641860 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "92f0a2b3-1573-415b-9672-a9a1b51afdda" (UID: "92f0a2b3-1573-415b-9672-a9a1b51afdda"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:50:23.643624 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.643591 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f0a2b3-1573-415b-9672-a9a1b51afdda-kube-api-access-7s6qh" (OuterVolumeSpecName: "kube-api-access-7s6qh") pod "92f0a2b3-1573-415b-9672-a9a1b51afdda" (UID: "92f0a2b3-1573-415b-9672-a9a1b51afdda"). InnerVolumeSpecName "kube-api-access-7s6qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:50:23.643718 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.643614 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "92f0a2b3-1573-415b-9672-a9a1b51afdda" (UID: "92f0a2b3-1573-415b-9672-a9a1b51afdda"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:50:23.643718 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.643673 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "92f0a2b3-1573-415b-9672-a9a1b51afdda" (UID: "92f0a2b3-1573-415b-9672-a9a1b51afdda"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:50:23.742399 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.742282 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-trusted-ca-bundle\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:23.742399 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.742315 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-oauth-config\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:23.742399 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.742328 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s6qh\" (UniqueName: \"kubernetes.io/projected/92f0a2b3-1573-415b-9672-a9a1b51afdda-kube-api-access-7s6qh\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:23.742399 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.742342 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-oauth-serving-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:23.742399 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.742353 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-serving-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:23.742399 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:23.742391 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/92f0a2b3-1573-415b-9672-a9a1b51afdda-console-config\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:24.020416 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:24.020311 2563 generic.go:356] "Generic (PLEG): container finished" podID="92f0a2b3-1573-415b-9672-a9a1b51afdda" containerID="bbe2d02bfff0e7bb71511923460c3252db1126eb4779c5d4ccdd913cc6c8e2fe" exitCode=0 Jul 29 08:50:24.020416 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:24.020374 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8bf6b87c-69xgb" Jul 29 08:50:24.020416 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:24.020400 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8bf6b87c-69xgb" event={"ID":"92f0a2b3-1573-415b-9672-a9a1b51afdda","Type":"ContainerDied","Data":"bbe2d02bfff0e7bb71511923460c3252db1126eb4779c5d4ccdd913cc6c8e2fe"} Jul 29 08:50:24.020642 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:24.020499 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8bf6b87c-69xgb" event={"ID":"92f0a2b3-1573-415b-9672-a9a1b51afdda","Type":"ContainerDied","Data":"b5eff873317300ef2f4a1219b4c486c1931a451e42b02ba2c0e88348df979992"} Jul 29 08:50:24.020642 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:24.020529 2563 scope.go:117] "RemoveContainer" containerID="bbe2d02bfff0e7bb71511923460c3252db1126eb4779c5d4ccdd913cc6c8e2fe" Jul 29 08:50:24.028910 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:24.028885 2563 scope.go:117] "RemoveContainer" containerID="bbe2d02bfff0e7bb71511923460c3252db1126eb4779c5d4ccdd913cc6c8e2fe" Jul 29 08:50:24.029194 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:50:24.029175 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe2d02bfff0e7bb71511923460c3252db1126eb4779c5d4ccdd913cc6c8e2fe\": container with ID starting with bbe2d02bfff0e7bb71511923460c3252db1126eb4779c5d4ccdd913cc6c8e2fe not found: ID does not exist" containerID="bbe2d02bfff0e7bb71511923460c3252db1126eb4779c5d4ccdd913cc6c8e2fe" Jul 29 08:50:24.029259 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:24.029203 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe2d02bfff0e7bb71511923460c3252db1126eb4779c5d4ccdd913cc6c8e2fe"} err="failed to get container status \"bbe2d02bfff0e7bb71511923460c3252db1126eb4779c5d4ccdd913cc6c8e2fe\": rpc error: code = NotFound desc = could not find container \"bbe2d02bfff0e7bb71511923460c3252db1126eb4779c5d4ccdd913cc6c8e2fe\": container with ID starting with bbe2d02bfff0e7bb71511923460c3252db1126eb4779c5d4ccdd913cc6c8e2fe not found: ID does not exist" Jul 29 08:50:24.060514 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:24.060477 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f8bf6b87c-69xgb"] Jul 29 08:50:24.071920 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:24.071895 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f8bf6b87c-69xgb"] Jul 29 08:50:25.360604 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:25.360566 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92f0a2b3-1573-415b-9672-a9a1b51afdda" path="/var/lib/kubelet/pods/92f0a2b3-1573-415b-9672-a9a1b51afdda/volumes" Jul 29 08:50:31.191728 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:31.191698 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:50:31.195537 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:31.195519 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-9b4dd9b9c-cxdrg" Jul 29 08:50:34.459757 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.459662 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b6d85bb57-6q42j" podUID="e6780c80-d143-457f-a6c0-3d36df807598" containerName="console" containerID="cri-o://c686c2b04a06879913d7312cd3755582588213f2c0a8af0841a24ba3b45f1384" gracePeriod=15 Jul 29 08:50:34.779465 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.779430 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:50:34.942116 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.942087 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-oauth-serving-cert\") pod \"e6780c80-d143-457f-a6c0-3d36df807598\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " Jul 29 08:50:34.942302 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.942147 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-console-config\") pod \"e6780c80-d143-457f-a6c0-3d36df807598\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " Jul 29 08:50:34.942302 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.942175 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6780c80-d143-457f-a6c0-3d36df807598-console-serving-cert\") pod \"e6780c80-d143-457f-a6c0-3d36df807598\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " Jul 29 08:50:34.942302 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.942210 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-service-ca\") pod \"e6780c80-d143-457f-a6c0-3d36df807598\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " Jul 29 08:50:34.942302 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.942234 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6780c80-d143-457f-a6c0-3d36df807598-console-oauth-config\") pod \"e6780c80-d143-457f-a6c0-3d36df807598\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " Jul 29 08:50:34.942302 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.942264 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-trusted-ca-bundle\") pod \"e6780c80-d143-457f-a6c0-3d36df807598\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " Jul 29 08:50:34.942588 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.942335 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-485dt\" (UniqueName: \"kubernetes.io/projected/e6780c80-d143-457f-a6c0-3d36df807598-kube-api-access-485dt\") pod \"e6780c80-d143-457f-a6c0-3d36df807598\" (UID: \"e6780c80-d143-457f-a6c0-3d36df807598\") " Jul 29 08:50:34.942588 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.942489 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e6780c80-d143-457f-a6c0-3d36df807598" (UID: "e6780c80-d143-457f-a6c0-3d36df807598"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:50:34.942588 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.942545 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-console-config" (OuterVolumeSpecName: "console-config") pod "e6780c80-d143-457f-a6c0-3d36df807598" (UID: "e6780c80-d143-457f-a6c0-3d36df807598"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:50:34.942733 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.942611 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-oauth-serving-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:34.942733 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.942632 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-console-config\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:34.942733 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.942624 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-service-ca" (OuterVolumeSpecName: "service-ca") pod "e6780c80-d143-457f-a6c0-3d36df807598" (UID: "e6780c80-d143-457f-a6c0-3d36df807598"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:50:34.942856 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.942834 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e6780c80-d143-457f-a6c0-3d36df807598" (UID: "e6780c80-d143-457f-a6c0-3d36df807598"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:50:34.944610 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.944577 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6780c80-d143-457f-a6c0-3d36df807598-kube-api-access-485dt" (OuterVolumeSpecName: "kube-api-access-485dt") pod "e6780c80-d143-457f-a6c0-3d36df807598" (UID: "e6780c80-d143-457f-a6c0-3d36df807598"). InnerVolumeSpecName "kube-api-access-485dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:50:34.944752 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.944735 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6780c80-d143-457f-a6c0-3d36df807598-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e6780c80-d143-457f-a6c0-3d36df807598" (UID: "e6780c80-d143-457f-a6c0-3d36df807598"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:50:34.944805 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:34.944749 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6780c80-d143-457f-a6c0-3d36df807598-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e6780c80-d143-457f-a6c0-3d36df807598" (UID: "e6780c80-d143-457f-a6c0-3d36df807598"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:50:35.043661 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:35.043584 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-485dt\" (UniqueName: \"kubernetes.io/projected/e6780c80-d143-457f-a6c0-3d36df807598-kube-api-access-485dt\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:35.043661 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:35.043624 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6780c80-d143-457f-a6c0-3d36df807598-console-serving-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:35.043661 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:35.043642 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:35.043661 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:35.043659 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e6780c80-d143-457f-a6c0-3d36df807598-console-oauth-config\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:35.043886 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:35.043675 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6780c80-d143-457f-a6c0-3d36df807598-trusted-ca-bundle\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:50:35.070047 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:35.070014 2563 generic.go:356] "Generic (PLEG): container finished" podID="e6780c80-d143-457f-a6c0-3d36df807598" containerID="c686c2b04a06879913d7312cd3755582588213f2c0a8af0841a24ba3b45f1384" exitCode=0 Jul 29 08:50:35.070200 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:35.070082 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6d85bb57-6q42j" Jul 29 08:50:35.070200 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:35.070098 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6d85bb57-6q42j" event={"ID":"e6780c80-d143-457f-a6c0-3d36df807598","Type":"ContainerDied","Data":"c686c2b04a06879913d7312cd3755582588213f2c0a8af0841a24ba3b45f1384"} Jul 29 08:50:35.070200 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:35.070154 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6d85bb57-6q42j" event={"ID":"e6780c80-d143-457f-a6c0-3d36df807598","Type":"ContainerDied","Data":"58f53f5924eb8c23f092b1e9974a314b439eab6c01630834e8bc670b957c28bb"} Jul 29 08:50:35.070200 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:35.070177 2563 scope.go:117] "RemoveContainer" containerID="c686c2b04a06879913d7312cd3755582588213f2c0a8af0841a24ba3b45f1384" Jul 29 08:50:35.078669 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:35.078648 2563 scope.go:117] "RemoveContainer" containerID="c686c2b04a06879913d7312cd3755582588213f2c0a8af0841a24ba3b45f1384" Jul 29 08:50:35.078964 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:50:35.078943 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c686c2b04a06879913d7312cd3755582588213f2c0a8af0841a24ba3b45f1384\": container with ID starting with c686c2b04a06879913d7312cd3755582588213f2c0a8af0841a24ba3b45f1384 not found: ID does not exist" containerID="c686c2b04a06879913d7312cd3755582588213f2c0a8af0841a24ba3b45f1384" Jul 29 08:50:35.079039 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:35.078977 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c686c2b04a06879913d7312cd3755582588213f2c0a8af0841a24ba3b45f1384"} err="failed to get container status \"c686c2b04a06879913d7312cd3755582588213f2c0a8af0841a24ba3b45f1384\": rpc error: code = NotFound desc = could not find container \"c686c2b04a06879913d7312cd3755582588213f2c0a8af0841a24ba3b45f1384\": container with ID starting with c686c2b04a06879913d7312cd3755582588213f2c0a8af0841a24ba3b45f1384 not found: ID does not exist" Jul 29 08:50:35.101801 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:35.101775 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b6d85bb57-6q42j"] Jul 29 08:50:35.116388 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:35.114399 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b6d85bb57-6q42j"] Jul 29 08:50:35.363175 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:50:35.363099 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6780c80-d143-457f-a6c0-3d36df807598" path="/var/lib/kubelet/pods/e6780c80-d143-457f-a6c0-3d36df807598/volumes" Jul 29 08:51:16.794410 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.794374 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ff75677f7-msp7b"] Jul 29 08:51:16.794904 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.794714 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af6b391f-8894-4505-a5c2-185f56251278" containerName="console" Jul 29 08:51:16.794904 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.794732 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="af6b391f-8894-4505-a5c2-185f56251278" containerName="console" Jul 29 08:51:16.794904 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.794753 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6780c80-d143-457f-a6c0-3d36df807598" containerName="console" Jul 29 08:51:16.794904 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.794759 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6780c80-d143-457f-a6c0-3d36df807598" containerName="console" Jul 29 08:51:16.794904 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.794771 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92f0a2b3-1573-415b-9672-a9a1b51afdda" containerName="console" Jul 29 08:51:16.794904 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.794777 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="92f0a2b3-1573-415b-9672-a9a1b51afdda" containerName="console" Jul 29 08:51:16.794904 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.794847 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="af6b391f-8894-4505-a5c2-185f56251278" containerName="console" Jul 29 08:51:16.794904 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.794859 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="92f0a2b3-1573-415b-9672-a9a1b51afdda" containerName="console" Jul 29 08:51:16.794904 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.794870 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6780c80-d143-457f-a6c0-3d36df807598" containerName="console" Jul 29 08:51:16.797356 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.797337 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.800230 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.800203 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 08:51:16.800341 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.800253 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 08:51:16.801701 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.801680 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 08:51:16.801806 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.801712 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 08:51:16.801988 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.801973 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fpl6s\"" Jul 29 08:51:16.803023 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.803003 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 08:51:16.806233 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.806215 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 08:51:16.809471 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.809448 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff75677f7-msp7b"] Jul 29 08:51:16.864158 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.864131 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-trusted-ca-bundle\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.864296 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.864176 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-oauth-config\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.864296 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.864271 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw5qr\" (UniqueName: \"kubernetes.io/projected/7af0153a-6647-4965-a4fb-d6223c22a7f7-kube-api-access-cw5qr\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.864430 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.864311 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-serving-cert\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.864430 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.864342 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-oauth-serving-cert\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.864503 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.864424 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-service-ca\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.864503 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.864479 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-config\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.965085 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.965054 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-config\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.965227 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.965094 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-trusted-ca-bundle\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.965227 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.965121 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-oauth-config\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.965227 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.965160 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw5qr\" (UniqueName: \"kubernetes.io/projected/7af0153a-6647-4965-a4fb-d6223c22a7f7-kube-api-access-cw5qr\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.965227 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.965188 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-serving-cert\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.965227 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.965212 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-oauth-serving-cert\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.965501 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.965393 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-service-ca\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.965952 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.965897 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-config\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.965952 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.965934 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-oauth-serving-cert\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.966086 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.966037 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-trusted-ca-bundle\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.966130 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.966110 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-service-ca\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.967732 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.967711 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-serving-cert\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.967795 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.967739 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-oauth-config\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:16.975603 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:16.975583 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw5qr\" (UniqueName: \"kubernetes.io/projected/7af0153a-6647-4965-a4fb-d6223c22a7f7-kube-api-access-cw5qr\") pod \"console-ff75677f7-msp7b\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:17.107264 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:17.107176 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:17.243508 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:17.243463 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff75677f7-msp7b"] Jul 29 08:51:17.246185 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:51:17.246161 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af0153a_6647_4965_a4fb_d6223c22a7f7.slice/crio-47a41bbbb156b4a695d5eff3a2b7ce82cf7dda8e8191af277c5af5c7fb7ea05c WatchSource:0}: Error finding container 47a41bbbb156b4a695d5eff3a2b7ce82cf7dda8e8191af277c5af5c7fb7ea05c: Status 404 returned error can't find the container with id 47a41bbbb156b4a695d5eff3a2b7ce82cf7dda8e8191af277c5af5c7fb7ea05c Jul 29 08:51:18.239090 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:18.239052 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff75677f7-msp7b" event={"ID":"7af0153a-6647-4965-a4fb-d6223c22a7f7","Type":"ContainerStarted","Data":"545f43ada381e5e438ae4cddacfa5eeb2184cf49d62c2a61b1a6b5491c606b67"} Jul 29 08:51:18.239592 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:18.239139 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff75677f7-msp7b" event={"ID":"7af0153a-6647-4965-a4fb-d6223c22a7f7","Type":"ContainerStarted","Data":"47a41bbbb156b4a695d5eff3a2b7ce82cf7dda8e8191af277c5af5c7fb7ea05c"} Jul 29 08:51:18.258100 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:18.258044 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ff75677f7-msp7b" podStartSLOduration=2.25802722 podStartE2EDuration="2.25802722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:51:18.256953256 +0000 UTC m=+191.381054056" watchObservedRunningTime="2026-07-29 08:51:18.25802722 +0000 UTC m=+191.382128019" Jul 29 08:51:27.108198 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:27.108150 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:27.108708 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:27.108238 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:27.112519 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:27.112496 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:51:27.278806 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:51:27.278777 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:52:10.113574 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.113542 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h"] Jul 29 08:52:10.115930 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.115910 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" Jul 29 08:52:10.118651 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.118629 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 08:52:10.119824 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.119803 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n4jml\"" Jul 29 08:52:10.119909 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.119836 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 08:52:10.125727 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.125705 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h"] Jul 29 08:52:10.204675 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.204638 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz4lg\" (UniqueName: \"kubernetes.io/projected/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-kube-api-access-hz4lg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h\" (UID: \"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" Jul 29 08:52:10.204819 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.204689 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h\" (UID: \"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" Jul 29 08:52:10.204819 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.204708 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h\" (UID: \"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" Jul 29 08:52:10.306133 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.306107 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz4lg\" (UniqueName: \"kubernetes.io/projected/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-kube-api-access-hz4lg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h\" (UID: \"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" Jul 29 08:52:10.306274 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.306146 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h\" (UID: \"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" Jul 29 08:52:10.306274 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.306165 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h\" (UID: \"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" Jul 29 08:52:10.306546 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.306528 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h\" (UID: \"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" Jul 29 08:52:10.306584 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.306546 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h\" (UID: \"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" Jul 29 08:52:10.315258 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.315235 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz4lg\" (UniqueName: \"kubernetes.io/projected/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-kube-api-access-hz4lg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h\" (UID: \"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" Jul 29 08:52:10.425527 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.425491 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" Jul 29 08:52:10.550532 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:10.550507 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h"] Jul 29 08:52:10.552474 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:52:10.552446 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc54bf29_d8b5_41cb_8b8b_47a6957dc03f.slice/crio-e5b52b7675de7407fae4f1a9782c3aeb6bbcbf0afa54f787aaa518828be8bb14 WatchSource:0}: Error finding container e5b52b7675de7407fae4f1a9782c3aeb6bbcbf0afa54f787aaa518828be8bb14: Status 404 returned error can't find the container with id e5b52b7675de7407fae4f1a9782c3aeb6bbcbf0afa54f787aaa518828be8bb14 Jul 29 08:52:11.438415 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:11.438373 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" event={"ID":"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f","Type":"ContainerStarted","Data":"e5b52b7675de7407fae4f1a9782c3aeb6bbcbf0afa54f787aaa518828be8bb14"} Jul 29 08:52:17.464509 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:17.464474 2563 generic.go:356] "Generic (PLEG): container finished" podID="fc54bf29-d8b5-41cb-8b8b-47a6957dc03f" containerID="076f4690e85feddac190d5c36b798ed5cfffbe4a5f1a54066fe0e8c9be5d0470" exitCode=0 Jul 29 08:52:17.464900 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:17.464538 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" event={"ID":"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f","Type":"ContainerDied","Data":"076f4690e85feddac190d5c36b798ed5cfffbe4a5f1a54066fe0e8c9be5d0470"} Jul 29 08:52:19.475040 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:19.475010 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" event={"ID":"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f","Type":"ContainerStarted","Data":"2505cc4bad39752d1baa9f0061a527a2a76f761ace618b684dc86e416e35e937"} Jul 29 08:52:20.480528 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:20.480494 2563 generic.go:356] "Generic (PLEG): container finished" podID="fc54bf29-d8b5-41cb-8b8b-47a6957dc03f" containerID="2505cc4bad39752d1baa9f0061a527a2a76f761ace618b684dc86e416e35e937" exitCode=0 Jul 29 08:52:20.480949 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:20.480525 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" event={"ID":"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f","Type":"ContainerDied","Data":"2505cc4bad39752d1baa9f0061a527a2a76f761ace618b684dc86e416e35e937"} Jul 29 08:52:27.511471 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:27.511433 2563 generic.go:356] "Generic (PLEG): container finished" podID="fc54bf29-d8b5-41cb-8b8b-47a6957dc03f" containerID="b37a01cbc3e2802f14e5248085f510f0b51150345fa025590c42957196c21168" exitCode=0 Jul 29 08:52:27.511856 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:27.511483 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" event={"ID":"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f","Type":"ContainerDied","Data":"b37a01cbc3e2802f14e5248085f510f0b51150345fa025590c42957196c21168"} Jul 29 08:52:28.642739 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:28.642717 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" Jul 29 08:52:28.667689 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:28.667663 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-util\") pod \"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f\" (UID: \"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f\") " Jul 29 08:52:28.667840 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:28.667714 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-bundle\") pod \"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f\" (UID: \"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f\") " Jul 29 08:52:28.667840 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:28.667754 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz4lg\" (UniqueName: \"kubernetes.io/projected/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-kube-api-access-hz4lg\") pod \"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f\" (UID: \"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f\") " Jul 29 08:52:28.668389 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:28.668343 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-bundle" (OuterVolumeSpecName: "bundle") pod "fc54bf29-d8b5-41cb-8b8b-47a6957dc03f" (UID: "fc54bf29-d8b5-41cb-8b8b-47a6957dc03f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:52:28.669990 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:28.669962 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-kube-api-access-hz4lg" (OuterVolumeSpecName: "kube-api-access-hz4lg") pod "fc54bf29-d8b5-41cb-8b8b-47a6957dc03f" (UID: "fc54bf29-d8b5-41cb-8b8b-47a6957dc03f"). InnerVolumeSpecName "kube-api-access-hz4lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:52:28.672425 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:28.672401 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-util" (OuterVolumeSpecName: "util") pod "fc54bf29-d8b5-41cb-8b8b-47a6957dc03f" (UID: "fc54bf29-d8b5-41cb-8b8b-47a6957dc03f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:52:28.768329 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:28.768238 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-bundle\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:52:28.768329 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:28.768275 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz4lg\" (UniqueName: \"kubernetes.io/projected/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-kube-api-access-hz4lg\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:52:28.768329 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:28.768287 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc54bf29-d8b5-41cb-8b8b-47a6957dc03f-util\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:52:29.520901 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:29.520875 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" Jul 29 08:52:29.520901 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:29.520888 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404846h" event={"ID":"fc54bf29-d8b5-41cb-8b8b-47a6957dc03f","Type":"ContainerDied","Data":"e5b52b7675de7407fae4f1a9782c3aeb6bbcbf0afa54f787aaa518828be8bb14"} Jul 29 08:52:29.521158 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:29.520931 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5b52b7675de7407fae4f1a9782c3aeb6bbcbf0afa54f787aaa518828be8bb14" Jul 29 08:52:40.145236 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.145202 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6rxqc"] Jul 29 08:52:40.145697 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.145573 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc54bf29-d8b5-41cb-8b8b-47a6957dc03f" containerName="pull" Jul 29 08:52:40.145697 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.145586 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc54bf29-d8b5-41cb-8b8b-47a6957dc03f" containerName="pull" Jul 29 08:52:40.145697 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.145599 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc54bf29-d8b5-41cb-8b8b-47a6957dc03f" containerName="util" Jul 29 08:52:40.145697 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.145604 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc54bf29-d8b5-41cb-8b8b-47a6957dc03f" containerName="util" Jul 29 08:52:40.145697 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.145612 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc54bf29-d8b5-41cb-8b8b-47a6957dc03f" containerName="extract" Jul 29 08:52:40.145697 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.145618 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc54bf29-d8b5-41cb-8b8b-47a6957dc03f" containerName="extract" Jul 29 08:52:40.145697 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.145676 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc54bf29-d8b5-41cb-8b8b-47a6957dc03f" containerName="extract" Jul 29 08:52:40.148497 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.148482 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6rxqc" Jul 29 08:52:40.151264 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.151245 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 29 08:52:40.152339 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.152319 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-f4qpn\"" Jul 29 08:52:40.152428 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.152376 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 29 08:52:40.158879 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.158859 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6rxqc"] Jul 29 08:52:40.257891 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.257862 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2441ac51-573f-4ec9-9b79-8cb61c44bb11-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6rxqc\" (UID: \"2441ac51-573f-4ec9-9b79-8cb61c44bb11\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6rxqc" Jul 29 08:52:40.258021 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.257893 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w667c\" (UniqueName: \"kubernetes.io/projected/2441ac51-573f-4ec9-9b79-8cb61c44bb11-kube-api-access-w667c\") pod \"cert-manager-cainjector-559cdd8c87-6rxqc\" (UID: \"2441ac51-573f-4ec9-9b79-8cb61c44bb11\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6rxqc" Jul 29 08:52:40.358640 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.358603 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2441ac51-573f-4ec9-9b79-8cb61c44bb11-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6rxqc\" (UID: \"2441ac51-573f-4ec9-9b79-8cb61c44bb11\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6rxqc" Jul 29 08:52:40.358640 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.358638 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w667c\" (UniqueName: \"kubernetes.io/projected/2441ac51-573f-4ec9-9b79-8cb61c44bb11-kube-api-access-w667c\") pod \"cert-manager-cainjector-559cdd8c87-6rxqc\" (UID: \"2441ac51-573f-4ec9-9b79-8cb61c44bb11\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6rxqc" Jul 29 08:52:40.367292 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.367266 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w667c\" (UniqueName: \"kubernetes.io/projected/2441ac51-573f-4ec9-9b79-8cb61c44bb11-kube-api-access-w667c\") pod \"cert-manager-cainjector-559cdd8c87-6rxqc\" (UID: \"2441ac51-573f-4ec9-9b79-8cb61c44bb11\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6rxqc" Jul 29 08:52:40.367292 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.367276 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2441ac51-573f-4ec9-9b79-8cb61c44bb11-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6rxqc\" (UID: \"2441ac51-573f-4ec9-9b79-8cb61c44bb11\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6rxqc" Jul 29 08:52:40.468847 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.468815 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6rxqc" Jul 29 08:52:40.594701 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:40.594670 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6rxqc"] Jul 29 08:52:40.597810 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:52:40.597784 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2441ac51_573f_4ec9_9b79_8cb61c44bb11.slice/crio-5962049024e114d431b7a80c9a8760836bb275d6bf56d34c4fd1a0d659422ab4 WatchSource:0}: Error finding container 5962049024e114d431b7a80c9a8760836bb275d6bf56d34c4fd1a0d659422ab4: Status 404 returned error can't find the container with id 5962049024e114d431b7a80c9a8760836bb275d6bf56d34c4fd1a0d659422ab4 Jul 29 08:52:41.574976 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:41.574937 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6rxqc" event={"ID":"2441ac51-573f-4ec9-9b79-8cb61c44bb11","Type":"ContainerStarted","Data":"5962049024e114d431b7a80c9a8760836bb275d6bf56d34c4fd1a0d659422ab4"} Jul 29 08:52:43.585275 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:43.585187 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6rxqc" event={"ID":"2441ac51-573f-4ec9-9b79-8cb61c44bb11","Type":"ContainerStarted","Data":"8329933fb5515c719b37335066c2889d17dd52f5281ac15057e22979d0683a06"} Jul 29 08:52:43.601161 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:43.601112 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6rxqc" podStartSLOduration=0.998020423 podStartE2EDuration="3.601095756s" totalImagesPullingTime="2.603075333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:52:43.600866835 +0000 UTC m=+276.724967635" watchObservedRunningTime="2026-07-29 08:52:43.601095756 +0000 UTC m=+276.725196556" Jul 29 08:52:45.685941 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:45.685905 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-npqmj"] Jul 29 08:52:45.689302 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:45.689280 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-npqmj" Jul 29 08:52:45.692340 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:45.692318 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-67n7p\"" Jul 29 08:52:45.700732 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:45.700711 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-npqmj"] Jul 29 08:52:45.804890 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:45.804858 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65lzn\" (UniqueName: \"kubernetes.io/projected/2ba2c131-6fc8-4d6b-807c-a8da626f3f7c-kube-api-access-65lzn\") pod \"cert-manager-webhook-6486ffd997-npqmj\" (UID: \"2ba2c131-6fc8-4d6b-807c-a8da626f3f7c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-npqmj" Jul 29 08:52:45.805021 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:45.804924 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ba2c131-6fc8-4d6b-807c-a8da626f3f7c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-npqmj\" (UID: \"2ba2c131-6fc8-4d6b-807c-a8da626f3f7c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-npqmj" Jul 29 08:52:45.905683 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:45.905652 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ba2c131-6fc8-4d6b-807c-a8da626f3f7c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-npqmj\" (UID: \"2ba2c131-6fc8-4d6b-807c-a8da626f3f7c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-npqmj" Jul 29 08:52:45.905789 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:45.905700 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65lzn\" (UniqueName: \"kubernetes.io/projected/2ba2c131-6fc8-4d6b-807c-a8da626f3f7c-kube-api-access-65lzn\") pod \"cert-manager-webhook-6486ffd997-npqmj\" (UID: \"2ba2c131-6fc8-4d6b-807c-a8da626f3f7c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-npqmj" Jul 29 08:52:45.914087 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:45.914063 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ba2c131-6fc8-4d6b-807c-a8da626f3f7c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-npqmj\" (UID: \"2ba2c131-6fc8-4d6b-807c-a8da626f3f7c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-npqmj" Jul 29 08:52:45.914184 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:45.914152 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65lzn\" (UniqueName: \"kubernetes.io/projected/2ba2c131-6fc8-4d6b-807c-a8da626f3f7c-kube-api-access-65lzn\") pod \"cert-manager-webhook-6486ffd997-npqmj\" (UID: \"2ba2c131-6fc8-4d6b-807c-a8da626f3f7c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-npqmj" Jul 29 08:52:45.997976 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:45.997900 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-npqmj" Jul 29 08:52:46.128458 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:46.128433 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-npqmj"] Jul 29 08:52:46.129978 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:52:46.129956 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ba2c131_6fc8_4d6b_807c_a8da626f3f7c.slice/crio-52eb39c6734f11895a511ec895f68eb1ebb89ea6e5423c1c3a6cfbbc29cd4d33 WatchSource:0}: Error finding container 52eb39c6734f11895a511ec895f68eb1ebb89ea6e5423c1c3a6cfbbc29cd4d33: Status 404 returned error can't find the container with id 52eb39c6734f11895a511ec895f68eb1ebb89ea6e5423c1c3a6cfbbc29cd4d33 Jul 29 08:52:46.598002 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:46.597970 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-npqmj" event={"ID":"2ba2c131-6fc8-4d6b-807c-a8da626f3f7c","Type":"ContainerStarted","Data":"e4126b9d6debd3adfcbafc6126f5f02b6347cbec695c33444a2f2caaf5127caa"} Jul 29 08:52:46.598172 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:46.598009 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-npqmj" event={"ID":"2ba2c131-6fc8-4d6b-807c-a8da626f3f7c","Type":"ContainerStarted","Data":"52eb39c6734f11895a511ec895f68eb1ebb89ea6e5423c1c3a6cfbbc29cd4d33"} Jul 29 08:52:46.598172 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:46.598036 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-npqmj" Jul 29 08:52:46.616777 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:46.616726 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-npqmj" podStartSLOduration=1.616710376 podStartE2EDuration="1.616710376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:52:46.615711309 +0000 UTC m=+279.739812109" watchObservedRunningTime="2026-07-29 08:52:46.616710376 +0000 UTC m=+279.740811174" Jul 29 08:52:48.761169 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:48.761131 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dwxtw"] Jul 29 08:52:48.763192 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:48.763171 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dwxtw" Jul 29 08:52:48.766085 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:48.766066 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jjk57\"" Jul 29 08:52:48.774568 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:48.774547 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dwxtw"] Jul 29 08:52:48.827214 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:48.827190 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glzx8\" (UniqueName: \"kubernetes.io/projected/c78a4c6d-0a6c-4663-a4a4-8d77ed3c6738-kube-api-access-glzx8\") pod \"cert-manager-55b4d954d5-dwxtw\" (UID: \"c78a4c6d-0a6c-4663-a4a4-8d77ed3c6738\") " pod="cert-manager/cert-manager-55b4d954d5-dwxtw" Jul 29 08:52:48.827344 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:48.827282 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c78a4c6d-0a6c-4663-a4a4-8d77ed3c6738-bound-sa-token\") pod \"cert-manager-55b4d954d5-dwxtw\" (UID: \"c78a4c6d-0a6c-4663-a4a4-8d77ed3c6738\") " pod="cert-manager/cert-manager-55b4d954d5-dwxtw" Jul 29 08:52:48.928530 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:48.928493 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glzx8\" (UniqueName: \"kubernetes.io/projected/c78a4c6d-0a6c-4663-a4a4-8d77ed3c6738-kube-api-access-glzx8\") pod \"cert-manager-55b4d954d5-dwxtw\" (UID: \"c78a4c6d-0a6c-4663-a4a4-8d77ed3c6738\") " pod="cert-manager/cert-manager-55b4d954d5-dwxtw" Jul 29 08:52:48.928635 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:48.928588 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c78a4c6d-0a6c-4663-a4a4-8d77ed3c6738-bound-sa-token\") pod \"cert-manager-55b4d954d5-dwxtw\" (UID: \"c78a4c6d-0a6c-4663-a4a4-8d77ed3c6738\") " pod="cert-manager/cert-manager-55b4d954d5-dwxtw" Jul 29 08:52:48.936865 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:48.936847 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c78a4c6d-0a6c-4663-a4a4-8d77ed3c6738-bound-sa-token\") pod \"cert-manager-55b4d954d5-dwxtw\" (UID: \"c78a4c6d-0a6c-4663-a4a4-8d77ed3c6738\") " pod="cert-manager/cert-manager-55b4d954d5-dwxtw" Jul 29 08:52:48.936943 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:48.936923 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glzx8\" (UniqueName: \"kubernetes.io/projected/c78a4c6d-0a6c-4663-a4a4-8d77ed3c6738-kube-api-access-glzx8\") pod \"cert-manager-55b4d954d5-dwxtw\" (UID: \"c78a4c6d-0a6c-4663-a4a4-8d77ed3c6738\") " pod="cert-manager/cert-manager-55b4d954d5-dwxtw" Jul 29 08:52:49.072202 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:49.072140 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dwxtw" Jul 29 08:52:49.202905 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:49.202871 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dwxtw"] Jul 29 08:52:49.206232 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:52:49.206197 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc78a4c6d_0a6c_4663_a4a4_8d77ed3c6738.slice/crio-874d7176d81d8a74e3b0a11a6f3372e4b4d054c962ce1a6362da2bebfa8db640 WatchSource:0}: Error finding container 874d7176d81d8a74e3b0a11a6f3372e4b4d054c962ce1a6362da2bebfa8db640: Status 404 returned error can't find the container with id 874d7176d81d8a74e3b0a11a6f3372e4b4d054c962ce1a6362da2bebfa8db640 Jul 29 08:52:49.611105 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:49.611057 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dwxtw" event={"ID":"c78a4c6d-0a6c-4663-a4a4-8d77ed3c6738","Type":"ContainerStarted","Data":"da960e8048745245d3cbbdecc51788c57881f7d0c8d479f85381b5386e8237fd"} Jul 29 08:52:49.611278 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:49.611114 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dwxtw" event={"ID":"c78a4c6d-0a6c-4663-a4a4-8d77ed3c6738","Type":"ContainerStarted","Data":"874d7176d81d8a74e3b0a11a6f3372e4b4d054c962ce1a6362da2bebfa8db640"} Jul 29 08:52:49.627729 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:49.627679 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dwxtw" podStartSLOduration=1.627662886 podStartE2EDuration="1.627662886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:52:49.62724321 +0000 UTC m=+282.751344010" watchObservedRunningTime="2026-07-29 08:52:49.627662886 +0000 UTC m=+282.751763687" Jul 29 08:52:52.605036 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:52:52.605007 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-npqmj" Jul 29 08:53:07.211177 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:07.211150 2563 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 08:53:08.760853 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:08.760813 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk"] Jul 29 08:53:08.763080 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:08.763060 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" Jul 29 08:53:08.766426 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:08.766408 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 08:53:08.766514 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:08.766452 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 08:53:08.768211 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:08.768195 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-n4jml\"" Jul 29 08:53:08.771927 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:08.771906 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk"] Jul 29 08:53:08.906243 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:08.906203 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d1a1a65-544e-4a4a-8f19-7a613d520072-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk\" (UID: \"8d1a1a65-544e-4a4a-8f19-7a613d520072\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" Jul 29 08:53:08.906430 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:08.906252 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d1a1a65-544e-4a4a-8f19-7a613d520072-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk\" (UID: \"8d1a1a65-544e-4a4a-8f19-7a613d520072\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" Jul 29 08:53:08.906430 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:08.906275 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9blf8\" (UniqueName: \"kubernetes.io/projected/8d1a1a65-544e-4a4a-8f19-7a613d520072-kube-api-access-9blf8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk\" (UID: \"8d1a1a65-544e-4a4a-8f19-7a613d520072\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" Jul 29 08:53:09.007599 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:09.007557 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d1a1a65-544e-4a4a-8f19-7a613d520072-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk\" (UID: \"8d1a1a65-544e-4a4a-8f19-7a613d520072\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" Jul 29 08:53:09.007787 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:09.007611 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d1a1a65-544e-4a4a-8f19-7a613d520072-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk\" (UID: \"8d1a1a65-544e-4a4a-8f19-7a613d520072\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" Jul 29 08:53:09.007787 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:09.007642 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9blf8\" (UniqueName: \"kubernetes.io/projected/8d1a1a65-544e-4a4a-8f19-7a613d520072-kube-api-access-9blf8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk\" (UID: \"8d1a1a65-544e-4a4a-8f19-7a613d520072\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" Jul 29 08:53:09.007963 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:09.007942 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d1a1a65-544e-4a4a-8f19-7a613d520072-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk\" (UID: \"8d1a1a65-544e-4a4a-8f19-7a613d520072\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" Jul 29 08:53:09.008103 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:09.007971 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d1a1a65-544e-4a4a-8f19-7a613d520072-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk\" (UID: \"8d1a1a65-544e-4a4a-8f19-7a613d520072\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" Jul 29 08:53:09.017079 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:09.017016 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9blf8\" (UniqueName: \"kubernetes.io/projected/8d1a1a65-544e-4a4a-8f19-7a613d520072-kube-api-access-9blf8\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk\" (UID: \"8d1a1a65-544e-4a4a-8f19-7a613d520072\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" Jul 29 08:53:09.072878 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:09.072850 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" Jul 29 08:53:09.205752 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:09.205728 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk"] Jul 29 08:53:09.207690 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:53:09.207663 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1a1a65_544e_4a4a_8f19_7a613d520072.slice/crio-b3d949be3542f409f81446941239ba08b7c8d959eb7e2188c5f78313d5e122d3 WatchSource:0}: Error finding container b3d949be3542f409f81446941239ba08b7c8d959eb7e2188c5f78313d5e122d3: Status 404 returned error can't find the container with id b3d949be3542f409f81446941239ba08b7c8d959eb7e2188c5f78313d5e122d3 Jul 29 08:53:09.690283 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:09.690248 2563 generic.go:356] "Generic (PLEG): container finished" podID="8d1a1a65-544e-4a4a-8f19-7a613d520072" containerID="1940accca8e331e6a7670dae64e782a38c5008ff614e8001534f26adb1d33d6f" exitCode=0 Jul 29 08:53:09.690471 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:09.690325 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" event={"ID":"8d1a1a65-544e-4a4a-8f19-7a613d520072","Type":"ContainerDied","Data":"1940accca8e331e6a7670dae64e782a38c5008ff614e8001534f26adb1d33d6f"} Jul 29 08:53:09.690471 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:09.690397 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" event={"ID":"8d1a1a65-544e-4a4a-8f19-7a613d520072","Type":"ContainerStarted","Data":"b3d949be3542f409f81446941239ba08b7c8d959eb7e2188c5f78313d5e122d3"} Jul 29 08:53:09.690879 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:09.690865 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:53:10.696153 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:10.696117 2563 generic.go:356] "Generic (PLEG): container finished" podID="8d1a1a65-544e-4a4a-8f19-7a613d520072" containerID="08d4cae7bc32108c5fd6beecb9ef1d4964cbbff94814e079598c04133eb86986" exitCode=0 Jul 29 08:53:10.696534 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:10.696186 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" event={"ID":"8d1a1a65-544e-4a4a-8f19-7a613d520072","Type":"ContainerDied","Data":"08d4cae7bc32108c5fd6beecb9ef1d4964cbbff94814e079598c04133eb86986"} Jul 29 08:53:11.701879 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:11.701840 2563 generic.go:356] "Generic (PLEG): container finished" podID="8d1a1a65-544e-4a4a-8f19-7a613d520072" containerID="43cf2360a7b880917caaa94a743c746de9aa80c91c84618dd12e17cd4d980680" exitCode=0 Jul 29 08:53:11.702255 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:11.701930 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" event={"ID":"8d1a1a65-544e-4a4a-8f19-7a613d520072","Type":"ContainerDied","Data":"43cf2360a7b880917caaa94a743c746de9aa80c91c84618dd12e17cd4d980680"} Jul 29 08:53:12.835604 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:12.835582 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" Jul 29 08:53:12.940865 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:12.940824 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d1a1a65-544e-4a4a-8f19-7a613d520072-util\") pod \"8d1a1a65-544e-4a4a-8f19-7a613d520072\" (UID: \"8d1a1a65-544e-4a4a-8f19-7a613d520072\") " Jul 29 08:53:12.941022 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:12.940877 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d1a1a65-544e-4a4a-8f19-7a613d520072-bundle\") pod \"8d1a1a65-544e-4a4a-8f19-7a613d520072\" (UID: \"8d1a1a65-544e-4a4a-8f19-7a613d520072\") " Jul 29 08:53:12.941022 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:12.940902 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9blf8\" (UniqueName: \"kubernetes.io/projected/8d1a1a65-544e-4a4a-8f19-7a613d520072-kube-api-access-9blf8\") pod \"8d1a1a65-544e-4a4a-8f19-7a613d520072\" (UID: \"8d1a1a65-544e-4a4a-8f19-7a613d520072\") " Jul 29 08:53:12.941655 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:12.941626 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1a1a65-544e-4a4a-8f19-7a613d520072-bundle" (OuterVolumeSpecName: "bundle") pod "8d1a1a65-544e-4a4a-8f19-7a613d520072" (UID: "8d1a1a65-544e-4a4a-8f19-7a613d520072"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:53:12.943040 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:12.943002 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1a1a65-544e-4a4a-8f19-7a613d520072-kube-api-access-9blf8" (OuterVolumeSpecName: "kube-api-access-9blf8") pod "8d1a1a65-544e-4a4a-8f19-7a613d520072" (UID: "8d1a1a65-544e-4a4a-8f19-7a613d520072"). InnerVolumeSpecName "kube-api-access-9blf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:53:12.947053 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:12.947026 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1a1a65-544e-4a4a-8f19-7a613d520072-util" (OuterVolumeSpecName: "util") pod "8d1a1a65-544e-4a4a-8f19-7a613d520072" (UID: "8d1a1a65-544e-4a4a-8f19-7a613d520072"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:53:13.041555 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:13.041467 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9blf8\" (UniqueName: \"kubernetes.io/projected/8d1a1a65-544e-4a4a-8f19-7a613d520072-kube-api-access-9blf8\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:53:13.041555 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:13.041502 2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d1a1a65-544e-4a4a-8f19-7a613d520072-util\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:53:13.041555 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:13.041511 2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d1a1a65-544e-4a4a-8f19-7a613d520072-bundle\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:53:13.713514 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:13.713487 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" Jul 29 08:53:13.713514 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:13.713498 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afd62vk" event={"ID":"8d1a1a65-544e-4a4a-8f19-7a613d520072","Type":"ContainerDied","Data":"b3d949be3542f409f81446941239ba08b7c8d959eb7e2188c5f78313d5e122d3"} Jul 29 08:53:13.713717 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:13.713539 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3d949be3542f409f81446941239ba08b7c8d959eb7e2188c5f78313d5e122d3" Jul 29 08:53:20.897158 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.897126 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-848779956-4vchn"] Jul 29 08:53:20.897557 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.897484 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1a1a65-544e-4a4a-8f19-7a613d520072" containerName="pull" Jul 29 08:53:20.897557 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.897503 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1a1a65-544e-4a4a-8f19-7a613d520072" containerName="pull" Jul 29 08:53:20.897557 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.897530 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1a1a65-544e-4a4a-8f19-7a613d520072" containerName="extract" Jul 29 08:53:20.897557 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.897540 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1a1a65-544e-4a4a-8f19-7a613d520072" containerName="extract" Jul 29 08:53:20.897557 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.897554 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1a1a65-544e-4a4a-8f19-7a613d520072" containerName="util" Jul 29 08:53:20.897738 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.897564 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1a1a65-544e-4a4a-8f19-7a613d520072" containerName="util" Jul 29 08:53:20.897738 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.897627 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d1a1a65-544e-4a4a-8f19-7a613d520072" containerName="extract" Jul 29 08:53:20.900902 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.900886 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:20.905047 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.905017 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 29 08:53:20.906458 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.906434 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 29 08:53:20.906458 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.906456 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 29 08:53:20.906458 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.906465 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 29 08:53:20.906693 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.906457 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 29 08:53:20.906693 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.906434 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8bqr6\"" Jul 29 08:53:20.916867 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:20.916846 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-848779956-4vchn"] Jul 29 08:53:21.013105 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:21.013070 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7de74cd6-0148-4fcb-b882-8ebb4abd4c01-metrics-cert\") pod \"lws-controller-manager-848779956-4vchn\" (UID: \"7de74cd6-0148-4fcb-b882-8ebb4abd4c01\") " pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:21.013291 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:21.013145 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7de74cd6-0148-4fcb-b882-8ebb4abd4c01-manager-config\") pod \"lws-controller-manager-848779956-4vchn\" (UID: \"7de74cd6-0148-4fcb-b882-8ebb4abd4c01\") " pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:21.013291 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:21.013172 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7de74cd6-0148-4fcb-b882-8ebb4abd4c01-cert\") pod \"lws-controller-manager-848779956-4vchn\" (UID: \"7de74cd6-0148-4fcb-b882-8ebb4abd4c01\") " pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:21.013291 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:21.013261 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj49s\" (UniqueName: \"kubernetes.io/projected/7de74cd6-0148-4fcb-b882-8ebb4abd4c01-kube-api-access-tj49s\") pod \"lws-controller-manager-848779956-4vchn\" (UID: \"7de74cd6-0148-4fcb-b882-8ebb4abd4c01\") " pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:21.114458 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:21.114417 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj49s\" (UniqueName: \"kubernetes.io/projected/7de74cd6-0148-4fcb-b882-8ebb4abd4c01-kube-api-access-tj49s\") pod \"lws-controller-manager-848779956-4vchn\" (UID: \"7de74cd6-0148-4fcb-b882-8ebb4abd4c01\") " pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:21.114665 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:21.114465 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7de74cd6-0148-4fcb-b882-8ebb4abd4c01-metrics-cert\") pod \"lws-controller-manager-848779956-4vchn\" (UID: \"7de74cd6-0148-4fcb-b882-8ebb4abd4c01\") " pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:21.114665 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:21.114530 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7de74cd6-0148-4fcb-b882-8ebb4abd4c01-manager-config\") pod \"lws-controller-manager-848779956-4vchn\" (UID: \"7de74cd6-0148-4fcb-b882-8ebb4abd4c01\") " pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:21.114665 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:21.114551 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7de74cd6-0148-4fcb-b882-8ebb4abd4c01-cert\") pod \"lws-controller-manager-848779956-4vchn\" (UID: \"7de74cd6-0148-4fcb-b882-8ebb4abd4c01\") " pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:21.115250 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:21.115225 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7de74cd6-0148-4fcb-b882-8ebb4abd4c01-manager-config\") pod \"lws-controller-manager-848779956-4vchn\" (UID: \"7de74cd6-0148-4fcb-b882-8ebb4abd4c01\") " pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:21.116978 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:21.116954 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7de74cd6-0148-4fcb-b882-8ebb4abd4c01-cert\") pod \"lws-controller-manager-848779956-4vchn\" (UID: \"7de74cd6-0148-4fcb-b882-8ebb4abd4c01\") " pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:21.117084 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:21.117066 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7de74cd6-0148-4fcb-b882-8ebb4abd4c01-metrics-cert\") pod \"lws-controller-manager-848779956-4vchn\" (UID: \"7de74cd6-0148-4fcb-b882-8ebb4abd4c01\") " pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:21.134344 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:21.134321 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj49s\" (UniqueName: \"kubernetes.io/projected/7de74cd6-0148-4fcb-b882-8ebb4abd4c01-kube-api-access-tj49s\") pod \"lws-controller-manager-848779956-4vchn\" (UID: \"7de74cd6-0148-4fcb-b882-8ebb4abd4c01\") " pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:21.209405 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:21.209381 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:21.352940 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:21.352909 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-848779956-4vchn"] Jul 29 08:53:21.356423 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:53:21.356352 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de74cd6_0148_4fcb_b882_8ebb4abd4c01.slice/crio-bd5c48fd5c7d6a5a025ec7bdd85e31ae109a5bc3addaa7635bb16ad1d044c625 WatchSource:0}: Error finding container bd5c48fd5c7d6a5a025ec7bdd85e31ae109a5bc3addaa7635bb16ad1d044c625: Status 404 returned error can't find the container with id bd5c48fd5c7d6a5a025ec7bdd85e31ae109a5bc3addaa7635bb16ad1d044c625 Jul 29 08:53:21.748017 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:21.747983 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" event={"ID":"7de74cd6-0148-4fcb-b882-8ebb4abd4c01","Type":"ContainerStarted","Data":"bd5c48fd5c7d6a5a025ec7bdd85e31ae109a5bc3addaa7635bb16ad1d044c625"} Jul 29 08:53:22.036460 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.036341 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd"] Jul 29 08:53:22.040545 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.040518 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.045304 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.045035 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 08:53:22.045304 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.045040 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 08:53:22.045304 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.045101 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 08:53:22.045304 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.045040 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 08:53:22.045304 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.045177 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7dtxx\"" Jul 29 08:53:22.073846 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.073819 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd"] Jul 29 08:53:22.223765 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.223718 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51537977-ebb1-460c-a77d-8292dc399306-tmp\") pod \"opendatahub-operator-controller-manager-67764bc7cd-7fsfd\" (UID: \"51537977-ebb1-460c-a77d-8292dc399306\") " pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.223959 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.223787 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/51537977-ebb1-460c-a77d-8292dc399306-manifests\") pod \"opendatahub-operator-controller-manager-67764bc7cd-7fsfd\" (UID: \"51537977-ebb1-460c-a77d-8292dc399306\") " pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.223959 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.223862 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/51537977-ebb1-460c-a77d-8292dc399306-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67764bc7cd-7fsfd\" (UID: \"51537977-ebb1-460c-a77d-8292dc399306\") " pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.223959 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.223920 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xm2c\" (UniqueName: \"kubernetes.io/projected/51537977-ebb1-460c-a77d-8292dc399306-kube-api-access-8xm2c\") pod \"opendatahub-operator-controller-manager-67764bc7cd-7fsfd\" (UID: \"51537977-ebb1-460c-a77d-8292dc399306\") " pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.223959 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.223949 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51537977-ebb1-460c-a77d-8292dc399306-webhook-cert\") pod \"opendatahub-operator-controller-manager-67764bc7cd-7fsfd\" (UID: \"51537977-ebb1-460c-a77d-8292dc399306\") " pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.326585 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.326211 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xm2c\" (UniqueName: \"kubernetes.io/projected/51537977-ebb1-460c-a77d-8292dc399306-kube-api-access-8xm2c\") pod \"opendatahub-operator-controller-manager-67764bc7cd-7fsfd\" (UID: \"51537977-ebb1-460c-a77d-8292dc399306\") " pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.326585 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.326264 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51537977-ebb1-460c-a77d-8292dc399306-webhook-cert\") pod \"opendatahub-operator-controller-manager-67764bc7cd-7fsfd\" (UID: \"51537977-ebb1-460c-a77d-8292dc399306\") " pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.326585 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.326393 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51537977-ebb1-460c-a77d-8292dc399306-tmp\") pod \"opendatahub-operator-controller-manager-67764bc7cd-7fsfd\" (UID: \"51537977-ebb1-460c-a77d-8292dc399306\") " pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.327395 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.327323 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/51537977-ebb1-460c-a77d-8292dc399306-manifests\") pod \"opendatahub-operator-controller-manager-67764bc7cd-7fsfd\" (UID: \"51537977-ebb1-460c-a77d-8292dc399306\") " pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.327515 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.327401 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/51537977-ebb1-460c-a77d-8292dc399306-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67764bc7cd-7fsfd\" (UID: \"51537977-ebb1-460c-a77d-8292dc399306\") " pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.333582 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.332710 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/51537977-ebb1-460c-a77d-8292dc399306-manifests\") pod \"opendatahub-operator-controller-manager-67764bc7cd-7fsfd\" (UID: \"51537977-ebb1-460c-a77d-8292dc399306\") " pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.334661 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.334599 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/51537977-ebb1-460c-a77d-8292dc399306-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67764bc7cd-7fsfd\" (UID: \"51537977-ebb1-460c-a77d-8292dc399306\") " pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.335535 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.335142 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51537977-ebb1-460c-a77d-8292dc399306-tmp\") pod \"opendatahub-operator-controller-manager-67764bc7cd-7fsfd\" (UID: \"51537977-ebb1-460c-a77d-8292dc399306\") " pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.337250 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.337228 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51537977-ebb1-460c-a77d-8292dc399306-webhook-cert\") pod \"opendatahub-operator-controller-manager-67764bc7cd-7fsfd\" (UID: \"51537977-ebb1-460c-a77d-8292dc399306\") " pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.343341 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.343321 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xm2c\" (UniqueName: \"kubernetes.io/projected/51537977-ebb1-460c-a77d-8292dc399306-kube-api-access-8xm2c\") pod \"opendatahub-operator-controller-manager-67764bc7cd-7fsfd\" (UID: \"51537977-ebb1-460c-a77d-8292dc399306\") " pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.356266 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.356233 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:22.536472 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.536450 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd"] Jul 29 08:53:22.538746 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:53:22.538718 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51537977_ebb1_460c_a77d_8292dc399306.slice/crio-c5dec48293fad489096b9d907d74c28c497d8420621a9f5b8d8db713e864c246 WatchSource:0}: Error finding container c5dec48293fad489096b9d907d74c28c497d8420621a9f5b8d8db713e864c246: Status 404 returned error can't find the container with id c5dec48293fad489096b9d907d74c28c497d8420621a9f5b8d8db713e864c246 Jul 29 08:53:22.754396 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:22.754341 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" event={"ID":"51537977-ebb1-460c-a77d-8292dc399306","Type":"ContainerStarted","Data":"c5dec48293fad489096b9d907d74c28c497d8420621a9f5b8d8db713e864c246"} Jul 29 08:53:24.768455 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:24.768388 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" event={"ID":"7de74cd6-0148-4fcb-b882-8ebb4abd4c01","Type":"ContainerStarted","Data":"e8eb56bbc9e6f485f41e5bf3349add89380dbd03a99635f53ecfffca15622a86"} Jul 29 08:53:24.768962 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:24.768582 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:24.793489 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:24.793345 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" podStartSLOduration=2.398298273 podStartE2EDuration="4.793329493s" totalImagesPullingTime="2.39503122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:53:24.792349536 +0000 UTC m=+317.916450336" watchObservedRunningTime="2026-07-29 08:53:24.793329493 +0000 UTC m=+317.917430292" Jul 29 08:53:26.779671 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:26.779626 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" event={"ID":"51537977-ebb1-460c-a77d-8292dc399306","Type":"ContainerStarted","Data":"f24a0b9e191f53cf6494e240b0c3a2c3329cc56f61070f63258580c8f5aa1b8e"} Jul 29 08:53:27.785502 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:27.785469 2563 generic.go:356] "Generic (PLEG): container finished" podID="51537977-ebb1-460c-a77d-8292dc399306" containerID="f24a0b9e191f53cf6494e240b0c3a2c3329cc56f61070f63258580c8f5aa1b8e" exitCode=0 Jul 29 08:53:27.785867 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:27.785559 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" event={"ID":"51537977-ebb1-460c-a77d-8292dc399306","Type":"ContainerDied","Data":"f24a0b9e191f53cf6494e240b0c3a2c3329cc56f61070f63258580c8f5aa1b8e"} Jul 29 08:53:28.792239 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:28.792200 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" event={"ID":"51537977-ebb1-460c-a77d-8292dc399306","Type":"ContainerStarted","Data":"de0e233cf2b72644a44da42487b335f20b95e85bb221cd4d8c25045fd1b1a53c"} Jul 29 08:53:28.792687 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:28.792339 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:28.816536 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:28.816482 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" podStartSLOduration=1.8275148300000001 podStartE2EDuration="6.81646743s" totalImagesPullingTime="3.931846438s" totalInitContainerRuntime="1.057106162s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:53:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:53:28.813844919 +0000 UTC m=+321.937945718" watchObservedRunningTime="2026-07-29 08:53:28.81646743 +0000 UTC m=+321.940568229" Jul 29 08:53:35.776602 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:35.776568 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-848779956-4vchn" Jul 29 08:53:39.799023 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:39.798990 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67764bc7cd-7fsfd" Jul 29 08:53:47.351087 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.351052 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc"] Jul 29 08:53:47.361217 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.361189 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc" Jul 29 08:53:47.372167 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.372146 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 29 08:53:47.374107 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.374087 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 29 08:53:47.374211 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.374087 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 08:53:47.375282 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.375265 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-v5hst\"" Jul 29 08:53:47.376805 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.376791 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 08:53:47.387616 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.387589 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc"] Jul 29 08:53:47.440005 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.439976 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c45b77a-8ead-4131-a581-123128f1df89-tls-certs\") pod \"kube-auth-proxy-7d544f58d6-r84vc\" (UID: \"8c45b77a-8ead-4131-a581-123128f1df89\") " pod="openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc" Jul 29 08:53:47.440158 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.440020 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfkkd\" (UniqueName: \"kubernetes.io/projected/8c45b77a-8ead-4131-a581-123128f1df89-kube-api-access-xfkkd\") pod \"kube-auth-proxy-7d544f58d6-r84vc\" (UID: \"8c45b77a-8ead-4131-a581-123128f1df89\") " pod="openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc" Jul 29 08:53:47.440158 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.440086 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c45b77a-8ead-4131-a581-123128f1df89-tmp\") pod \"kube-auth-proxy-7d544f58d6-r84vc\" (UID: \"8c45b77a-8ead-4131-a581-123128f1df89\") " pod="openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc" Jul 29 08:53:47.541289 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.541257 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c45b77a-8ead-4131-a581-123128f1df89-tls-certs\") pod \"kube-auth-proxy-7d544f58d6-r84vc\" (UID: \"8c45b77a-8ead-4131-a581-123128f1df89\") " pod="openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc" Jul 29 08:53:47.541492 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.541293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfkkd\" (UniqueName: \"kubernetes.io/projected/8c45b77a-8ead-4131-a581-123128f1df89-kube-api-access-xfkkd\") pod \"kube-auth-proxy-7d544f58d6-r84vc\" (UID: \"8c45b77a-8ead-4131-a581-123128f1df89\") " pod="openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc" Jul 29 08:53:47.541492 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.541318 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c45b77a-8ead-4131-a581-123128f1df89-tmp\") pod \"kube-auth-proxy-7d544f58d6-r84vc\" (UID: \"8c45b77a-8ead-4131-a581-123128f1df89\") " pod="openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc" Jul 29 08:53:47.541492 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:53:47.541445 2563 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 29 08:53:47.541664 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:53:47.541539 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c45b77a-8ead-4131-a581-123128f1df89-tls-certs podName:8c45b77a-8ead-4131-a581-123128f1df89 nodeName:}" failed. No retries permitted until 2026-07-29 08:53:48.04151643 +0000 UTC m=+341.165617231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8c45b77a-8ead-4131-a581-123128f1df89-tls-certs") pod "kube-auth-proxy-7d544f58d6-r84vc" (UID: "8c45b77a-8ead-4131-a581-123128f1df89") : secret "kube-auth-proxy-tls" not found Jul 29 08:53:47.543481 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.543463 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c45b77a-8ead-4131-a581-123128f1df89-tmp\") pod \"kube-auth-proxy-7d544f58d6-r84vc\" (UID: \"8c45b77a-8ead-4131-a581-123128f1df89\") " pod="openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc" Jul 29 08:53:47.562038 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:47.562008 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfkkd\" (UniqueName: \"kubernetes.io/projected/8c45b77a-8ead-4131-a581-123128f1df89-kube-api-access-xfkkd\") pod \"kube-auth-proxy-7d544f58d6-r84vc\" (UID: \"8c45b77a-8ead-4131-a581-123128f1df89\") " pod="openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc" Jul 29 08:53:48.046713 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:48.046679 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c45b77a-8ead-4131-a581-123128f1df89-tls-certs\") pod \"kube-auth-proxy-7d544f58d6-r84vc\" (UID: \"8c45b77a-8ead-4131-a581-123128f1df89\") " pod="openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc" Jul 29 08:53:48.049267 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:48.049232 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c45b77a-8ead-4131-a581-123128f1df89-tls-certs\") pod \"kube-auth-proxy-7d544f58d6-r84vc\" (UID: \"8c45b77a-8ead-4131-a581-123128f1df89\") " pod="openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc" Jul 29 08:53:48.270916 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:48.270881 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc" Jul 29 08:53:48.401842 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:48.401817 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc"] Jul 29 08:53:48.404156 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:53:48.404132 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c45b77a_8ead_4131_a581_123128f1df89.slice/crio-936a308bcb451684bae132ccd6299f51f4c2296478251ff7582f690c350585b4 WatchSource:0}: Error finding container 936a308bcb451684bae132ccd6299f51f4c2296478251ff7582f690c350585b4: Status 404 returned error can't find the container with id 936a308bcb451684bae132ccd6299f51f4c2296478251ff7582f690c350585b4 Jul 29 08:53:48.886376 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:48.886324 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc" event={"ID":"8c45b77a-8ead-4131-a581-123128f1df89","Type":"ContainerStarted","Data":"936a308bcb451684bae132ccd6299f51f4c2296478251ff7582f690c350585b4"} Jul 29 08:53:51.902403 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:51.902335 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc" event={"ID":"8c45b77a-8ead-4131-a581-123128f1df89","Type":"ContainerStarted","Data":"83414f84b96ab568afa437d8d8adff7839d4ce8614d9a02bbc2f6b547035ee95"} Jul 29 08:53:51.922700 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:53:51.920733 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d544f58d6-r84vc" podStartSLOduration=2.434985014 podStartE2EDuration="4.920705359s" totalImagesPullingTime="2.485720345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:53:51.918002588 +0000 UTC m=+345.042103389" watchObservedRunningTime="2026-07-29 08:53:51.920705359 +0000 UTC m=+345.044806159" Jul 29 08:55:11.788677 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:11.788647 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff75677f7-msp7b"] Jul 29 08:55:36.813449 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:36.813327 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-ff75677f7-msp7b" podUID="7af0153a-6647-4965-a4fb-d6223c22a7f7" containerName="console" containerID="cri-o://545f43ada381e5e438ae4cddacfa5eeb2184cf49d62c2a61b1a6b5491c606b67" gracePeriod=15 Jul 29 08:55:37.052127 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.052104 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:55:37.077927 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.077860 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-oauth-serving-cert\") pod \"7af0153a-6647-4965-a4fb-d6223c22a7f7\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " Jul 29 08:55:37.077927 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.077917 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-oauth-config\") pod \"7af0153a-6647-4965-a4fb-d6223c22a7f7\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " Jul 29 08:55:37.078112 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.077955 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-trusted-ca-bundle\") pod \"7af0153a-6647-4965-a4fb-d6223c22a7f7\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " Jul 29 08:55:37.078112 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.078035 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-serving-cert\") pod \"7af0153a-6647-4965-a4fb-d6223c22a7f7\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " Jul 29 08:55:37.078112 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.078076 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-service-ca\") pod \"7af0153a-6647-4965-a4fb-d6223c22a7f7\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " Jul 29 08:55:37.078112 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.078105 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-config\") pod \"7af0153a-6647-4965-a4fb-d6223c22a7f7\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " Jul 29 08:55:37.078303 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.078154 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw5qr\" (UniqueName: \"kubernetes.io/projected/7af0153a-6647-4965-a4fb-d6223c22a7f7-kube-api-access-cw5qr\") pod \"7af0153a-6647-4965-a4fb-d6223c22a7f7\" (UID: \"7af0153a-6647-4965-a4fb-d6223c22a7f7\") " Jul 29 08:55:37.078303 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.078241 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7af0153a-6647-4965-a4fb-d6223c22a7f7" (UID: "7af0153a-6647-4965-a4fb-d6223c22a7f7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:55:37.078475 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.078310 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7af0153a-6647-4965-a4fb-d6223c22a7f7" (UID: "7af0153a-6647-4965-a4fb-d6223c22a7f7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:55:37.078605 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.078568 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-service-ca" (OuterVolumeSpecName: "service-ca") pod "7af0153a-6647-4965-a4fb-d6223c22a7f7" (UID: "7af0153a-6647-4965-a4fb-d6223c22a7f7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:55:37.078665 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.078610 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-config" (OuterVolumeSpecName: "console-config") pod "7af0153a-6647-4965-a4fb-d6223c22a7f7" (UID: "7af0153a-6647-4965-a4fb-d6223c22a7f7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 08:55:37.078708 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.078664 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-oauth-serving-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:55:37.078708 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.078682 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-trusted-ca-bundle\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:55:37.078708 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.078695 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:55:37.087487 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.087447 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af0153a-6647-4965-a4fb-d6223c22a7f7-kube-api-access-cw5qr" (OuterVolumeSpecName: "kube-api-access-cw5qr") pod "7af0153a-6647-4965-a4fb-d6223c22a7f7" (UID: "7af0153a-6647-4965-a4fb-d6223c22a7f7"). InnerVolumeSpecName "kube-api-access-cw5qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:55:37.087606 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.087492 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7af0153a-6647-4965-a4fb-d6223c22a7f7" (UID: "7af0153a-6647-4965-a4fb-d6223c22a7f7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:55:37.087606 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.087562 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7af0153a-6647-4965-a4fb-d6223c22a7f7" (UID: "7af0153a-6647-4965-a4fb-d6223c22a7f7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 08:55:37.179836 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.179805 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-serving-cert\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:55:37.179836 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.179832 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-config\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:55:37.179836 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.179842 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw5qr\" (UniqueName: \"kubernetes.io/projected/7af0153a-6647-4965-a4fb-d6223c22a7f7-kube-api-access-cw5qr\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:55:37.180052 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.179852 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7af0153a-6647-4965-a4fb-d6223c22a7f7-console-oauth-config\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:55:37.382169 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.382139 2563 generic.go:356] "Generic (PLEG): container finished" podID="7af0153a-6647-4965-a4fb-d6223c22a7f7" containerID="545f43ada381e5e438ae4cddacfa5eeb2184cf49d62c2a61b1a6b5491c606b67" exitCode=0 Jul 29 08:55:37.382296 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.382186 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff75677f7-msp7b" Jul 29 08:55:37.382296 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.382213 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff75677f7-msp7b" event={"ID":"7af0153a-6647-4965-a4fb-d6223c22a7f7","Type":"ContainerDied","Data":"545f43ada381e5e438ae4cddacfa5eeb2184cf49d62c2a61b1a6b5491c606b67"} Jul 29 08:55:37.382296 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.382260 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff75677f7-msp7b" event={"ID":"7af0153a-6647-4965-a4fb-d6223c22a7f7","Type":"ContainerDied","Data":"47a41bbbb156b4a695d5eff3a2b7ce82cf7dda8e8191af277c5af5c7fb7ea05c"} Jul 29 08:55:37.382296 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.382287 2563 scope.go:117] "RemoveContainer" containerID="545f43ada381e5e438ae4cddacfa5eeb2184cf49d62c2a61b1a6b5491c606b67" Jul 29 08:55:37.390821 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.390806 2563 scope.go:117] "RemoveContainer" containerID="545f43ada381e5e438ae4cddacfa5eeb2184cf49d62c2a61b1a6b5491c606b67" Jul 29 08:55:37.391069 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:55:37.391052 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545f43ada381e5e438ae4cddacfa5eeb2184cf49d62c2a61b1a6b5491c606b67\": container with ID starting with 545f43ada381e5e438ae4cddacfa5eeb2184cf49d62c2a61b1a6b5491c606b67 not found: ID does not exist" containerID="545f43ada381e5e438ae4cddacfa5eeb2184cf49d62c2a61b1a6b5491c606b67" Jul 29 08:55:37.391151 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.391076 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545f43ada381e5e438ae4cddacfa5eeb2184cf49d62c2a61b1a6b5491c606b67"} err="failed to get container status \"545f43ada381e5e438ae4cddacfa5eeb2184cf49d62c2a61b1a6b5491c606b67\": rpc error: code = NotFound desc = could not find container \"545f43ada381e5e438ae4cddacfa5eeb2184cf49d62c2a61b1a6b5491c606b67\": container with ID starting with 545f43ada381e5e438ae4cddacfa5eeb2184cf49d62c2a61b1a6b5491c606b67 not found: ID does not exist" Jul 29 08:55:37.399793 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.399771 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff75677f7-msp7b"] Jul 29 08:55:37.403666 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:37.403644 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ff75677f7-msp7b"] Jul 29 08:55:39.361467 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:39.361431 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7af0153a-6647-4965-a4fb-d6223c22a7f7" path="/var/lib/kubelet/pods/7af0153a-6647-4965-a4fb-d6223c22a7f7/volumes" Jul 29 08:55:43.011269 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.011234 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf"] Jul 29 08:55:43.011674 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.011566 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af0153a-6647-4965-a4fb-d6223c22a7f7" containerName="console" Jul 29 08:55:43.011674 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.011577 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af0153a-6647-4965-a4fb-d6223c22a7f7" containerName="console" Jul 29 08:55:43.011674 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.011632 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7af0153a-6647-4965-a4fb-d6223c22a7f7" containerName="console" Jul 29 08:55:43.015892 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.015874 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" Jul 29 08:55:43.022254 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.022140 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qsh69\"" Jul 29 08:55:43.022254 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.022157 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 08:55:43.022423 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.022412 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 08:55:43.032070 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.032050 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf"] Jul 29 08:55:43.128480 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.128441 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxtzn\" (UniqueName: \"kubernetes.io/projected/fb519a7a-6863-4f9a-baa8-33b81f426a95-kube-api-access-qxtzn\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-smqmf\" (UID: \"fb519a7a-6863-4f9a-baa8-33b81f426a95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" Jul 29 08:55:43.128644 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.128505 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb519a7a-6863-4f9a-baa8-33b81f426a95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-smqmf\" (UID: \"fb519a7a-6863-4f9a-baa8-33b81f426a95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" Jul 29 08:55:43.229213 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.229178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxtzn\" (UniqueName: \"kubernetes.io/projected/fb519a7a-6863-4f9a-baa8-33b81f426a95-kube-api-access-qxtzn\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-smqmf\" (UID: \"fb519a7a-6863-4f9a-baa8-33b81f426a95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" Jul 29 08:55:43.229392 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.229239 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb519a7a-6863-4f9a-baa8-33b81f426a95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-smqmf\" (UID: \"fb519a7a-6863-4f9a-baa8-33b81f426a95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" Jul 29 08:55:43.229593 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.229578 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb519a7a-6863-4f9a-baa8-33b81f426a95-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-smqmf\" (UID: \"fb519a7a-6863-4f9a-baa8-33b81f426a95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" Jul 29 08:55:43.245925 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.245901 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxtzn\" (UniqueName: \"kubernetes.io/projected/fb519a7a-6863-4f9a-baa8-33b81f426a95-kube-api-access-qxtzn\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-smqmf\" (UID: \"fb519a7a-6863-4f9a-baa8-33b81f426a95\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" Jul 29 08:55:43.326210 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.326119 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" Jul 29 08:55:43.463005 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.462984 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf"] Jul 29 08:55:43.465647 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:55:43.465621 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb519a7a_6863_4f9a_baa8_33b81f426a95.slice/crio-e9fb337b9a5806df5f83a460edbb02b375eb9ae525d9495b1ea281e0ccc4830e WatchSource:0}: Error finding container e9fb337b9a5806df5f83a460edbb02b375eb9ae525d9495b1ea281e0ccc4830e: Status 404 returned error can't find the container with id e9fb337b9a5806df5f83a460edbb02b375eb9ae525d9495b1ea281e0ccc4830e Jul 29 08:55:43.693826 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.693791 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf"] Jul 29 08:55:43.701109 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.701080 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf"] Jul 29 08:55:43.739677 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.739642 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv"] Jul 29 08:55:43.745196 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.745172 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" Jul 29 08:55:43.761062 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.761038 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv"] Jul 29 08:55:43.836024 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.835993 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pnfsv\" (UID: \"e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" Jul 29 08:55:43.836216 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.836028 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfkct\" (UniqueName: \"kubernetes.io/projected/e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15-kube-api-access-pfkct\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pnfsv\" (UID: \"e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" Jul 29 08:55:43.937466 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.937428 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pnfsv\" (UID: \"e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" Jul 29 08:55:43.937635 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.937471 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfkct\" (UniqueName: \"kubernetes.io/projected/e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15-kube-api-access-pfkct\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pnfsv\" (UID: \"e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" Jul 29 08:55:43.937879 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.937858 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pnfsv\" (UID: \"e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" Jul 29 08:55:43.951224 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:43.951160 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfkct\" (UniqueName: \"kubernetes.io/projected/e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15-kube-api-access-pfkct\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-pnfsv\" (UID: \"e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" Jul 29 08:55:44.056865 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:44.056833 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" Jul 29 08:55:44.195146 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:44.195122 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv"] Jul 29 08:55:44.212830 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:55:44.212768 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b7f1f9_2d6b_463b_babe_fe4ec3a5af15.slice/crio-fa6d33d254cd7a4e0fecf92453817d7a8e80cdc28fb27df2d48882d11b67c16b WatchSource:0}: Error finding container fa6d33d254cd7a4e0fecf92453817d7a8e80cdc28fb27df2d48882d11b67c16b: Status 404 returned error can't find the container with id fa6d33d254cd7a4e0fecf92453817d7a8e80cdc28fb27df2d48882d11b67c16b Jul 29 08:55:44.415653 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:44.415620 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" event={"ID":"e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15","Type":"ContainerStarted","Data":"fa6d33d254cd7a4e0fecf92453817d7a8e80cdc28fb27df2d48882d11b67c16b"} Jul 29 08:55:48.441943 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:48.441879 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" podUID="fb519a7a-6863-4f9a-baa8-33b81f426a95" containerName="manager" containerID="cri-o://aedabab150cfd66ca92b913f85c9465f8803848f50599b165c4af198713531e2" gracePeriod=2 Jul 29 08:55:48.443511 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:48.443479 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" event={"ID":"e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15","Type":"ContainerStarted","Data":"1d1acf6805bd4e8073c12047295f2308b62b9ac89f8e0f337f246a244a62c84a"} Jul 29 08:55:48.443638 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:48.443583 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" Jul 29 08:55:48.444851 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:55:48.444820 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-smqmf\" is forbidden: User \"system:node:ip-10-0-141-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-66.ec2.internal' and this object" podUID="fb519a7a-6863-4f9a-baa8-33b81f426a95" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" Jul 29 08:55:48.447214 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:55:48.447190 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-smqmf\" is forbidden: User \"system:node:ip-10-0-141-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-66.ec2.internal' and this object" podUID="fb519a7a-6863-4f9a-baa8-33b81f426a95" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" Jul 29 08:55:48.468106 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:48.468050 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" podStartSLOduration=1.927461876 podStartE2EDuration="5.468031793s" totalImagesPullingTime="3.540569917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:55:48.466073007 +0000 UTC m=+461.590173808" watchObservedRunningTime="2026-07-29 08:55:48.468031793 +0000 UTC m=+461.592132592" Jul 29 08:55:48.676456 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:48.676429 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" Jul 29 08:55:48.679160 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:55:48.679121 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-smqmf\" is forbidden: User \"system:node:ip-10-0-141-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-66.ec2.internal' and this object" podUID="fb519a7a-6863-4f9a-baa8-33b81f426a95" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" Jul 29 08:55:48.785176 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:48.785100 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxtzn\" (UniqueName: \"kubernetes.io/projected/fb519a7a-6863-4f9a-baa8-33b81f426a95-kube-api-access-qxtzn\") pod \"fb519a7a-6863-4f9a-baa8-33b81f426a95\" (UID: \"fb519a7a-6863-4f9a-baa8-33b81f426a95\") " Jul 29 08:55:48.785176 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:48.785166 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb519a7a-6863-4f9a-baa8-33b81f426a95-extensions-socket-volume\") pod \"fb519a7a-6863-4f9a-baa8-33b81f426a95\" (UID: \"fb519a7a-6863-4f9a-baa8-33b81f426a95\") " Jul 29 08:55:48.785478 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:48.785448 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb519a7a-6863-4f9a-baa8-33b81f426a95-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fb519a7a-6863-4f9a-baa8-33b81f426a95" (UID: "fb519a7a-6863-4f9a-baa8-33b81f426a95"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:55:48.787325 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:48.787304 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb519a7a-6863-4f9a-baa8-33b81f426a95-kube-api-access-qxtzn" (OuterVolumeSpecName: "kube-api-access-qxtzn") pod "fb519a7a-6863-4f9a-baa8-33b81f426a95" (UID: "fb519a7a-6863-4f9a-baa8-33b81f426a95"). InnerVolumeSpecName "kube-api-access-qxtzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:55:48.886583 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:48.886548 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb519a7a-6863-4f9a-baa8-33b81f426a95-extensions-socket-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:55:48.886583 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:48.886581 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxtzn\" (UniqueName: \"kubernetes.io/projected/fb519a7a-6863-4f9a-baa8-33b81f426a95-kube-api-access-qxtzn\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:55:49.361817 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:49.361787 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb519a7a-6863-4f9a-baa8-33b81f426a95" path="/var/lib/kubelet/pods/fb519a7a-6863-4f9a-baa8-33b81f426a95/volumes" Jul 29 08:55:49.449610 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:49.449577 2563 generic.go:356] "Generic (PLEG): container finished" podID="fb519a7a-6863-4f9a-baa8-33b81f426a95" containerID="aedabab150cfd66ca92b913f85c9465f8803848f50599b165c4af198713531e2" exitCode=2 Jul 29 08:55:49.450043 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:49.449623 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" Jul 29 08:55:49.450043 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:49.449655 2563 scope.go:117] "RemoveContainer" containerID="aedabab150cfd66ca92b913f85c9465f8803848f50599b165c4af198713531e2" Jul 29 08:55:49.452388 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:55:49.452350 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-smqmf\" is forbidden: User \"system:node:ip-10-0-141-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-66.ec2.internal' and this object" podUID="fb519a7a-6863-4f9a-baa8-33b81f426a95" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" Jul 29 08:55:49.454798 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:55:49.454771 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-smqmf\" is forbidden: User \"system:node:ip-10-0-141-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-66.ec2.internal' and this object" podUID="fb519a7a-6863-4f9a-baa8-33b81f426a95" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-smqmf" Jul 29 08:55:49.458390 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:49.458352 2563 scope.go:117] "RemoveContainer" containerID="aedabab150cfd66ca92b913f85c9465f8803848f50599b165c4af198713531e2" Jul 29 08:55:49.458655 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:55:49.458637 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aedabab150cfd66ca92b913f85c9465f8803848f50599b165c4af198713531e2\": container with ID starting with aedabab150cfd66ca92b913f85c9465f8803848f50599b165c4af198713531e2 not found: ID does not exist" containerID="aedabab150cfd66ca92b913f85c9465f8803848f50599b165c4af198713531e2" Jul 29 08:55:49.458701 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:49.458666 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aedabab150cfd66ca92b913f85c9465f8803848f50599b165c4af198713531e2"} err="failed to get container status \"aedabab150cfd66ca92b913f85c9465f8803848f50599b165c4af198713531e2\": rpc error: code = NotFound desc = could not find container \"aedabab150cfd66ca92b913f85c9465f8803848f50599b165c4af198713531e2\": container with ID starting with aedabab150cfd66ca92b913f85c9465f8803848f50599b165c4af198713531e2 not found: ID does not exist" Jul 29 08:55:59.452849 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:55:59.452815 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" Jul 29 08:56:15.752754 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:15.752715 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv"] Jul 29 08:56:15.753200 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:15.752970 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" podUID="e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15" containerName="manager" containerID="cri-o://1d1acf6805bd4e8073c12047295f2308b62b9ac89f8e0f337f246a244a62c84a" gracePeriod=10 Jul 29 08:56:16.011854 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.011789 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" Jul 29 08:56:16.113551 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.113517 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15-extensions-socket-volume\") pod \"e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15\" (UID: \"e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15\") " Jul 29 08:56:16.113714 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.113581 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfkct\" (UniqueName: \"kubernetes.io/projected/e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15-kube-api-access-pfkct\") pod \"e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15\" (UID: \"e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15\") " Jul 29 08:56:16.113940 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.113920 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15" (UID: "e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 08:56:16.115776 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.115756 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15-kube-api-access-pfkct" (OuterVolumeSpecName: "kube-api-access-pfkct") pod "e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15" (UID: "e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15"). InnerVolumeSpecName "kube-api-access-pfkct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:56:16.214305 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.214274 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15-extensions-socket-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:56:16.214305 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.214300 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfkct\" (UniqueName: \"kubernetes.io/projected/e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15-kube-api-access-pfkct\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:56:16.579868 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.579831 2563 generic.go:356] "Generic (PLEG): container finished" podID="e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15" containerID="1d1acf6805bd4e8073c12047295f2308b62b9ac89f8e0f337f246a244a62c84a" exitCode=0 Jul 29 08:56:16.579868 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.579871 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" Jul 29 08:56:16.580067 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.579892 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" event={"ID":"e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15","Type":"ContainerDied","Data":"1d1acf6805bd4e8073c12047295f2308b62b9ac89f8e0f337f246a244a62c84a"} Jul 29 08:56:16.580067 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.579942 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv" event={"ID":"e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15","Type":"ContainerDied","Data":"fa6d33d254cd7a4e0fecf92453817d7a8e80cdc28fb27df2d48882d11b67c16b"} Jul 29 08:56:16.580067 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.579968 2563 scope.go:117] "RemoveContainer" containerID="1d1acf6805bd4e8073c12047295f2308b62b9ac89f8e0f337f246a244a62c84a" Jul 29 08:56:16.588496 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.588479 2563 scope.go:117] "RemoveContainer" containerID="1d1acf6805bd4e8073c12047295f2308b62b9ac89f8e0f337f246a244a62c84a" Jul 29 08:56:16.588725 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:56:16.588709 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1acf6805bd4e8073c12047295f2308b62b9ac89f8e0f337f246a244a62c84a\": container with ID starting with 1d1acf6805bd4e8073c12047295f2308b62b9ac89f8e0f337f246a244a62c84a not found: ID does not exist" containerID="1d1acf6805bd4e8073c12047295f2308b62b9ac89f8e0f337f246a244a62c84a" Jul 29 08:56:16.588771 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.588732 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1acf6805bd4e8073c12047295f2308b62b9ac89f8e0f337f246a244a62c84a"} err="failed to get container status \"1d1acf6805bd4e8073c12047295f2308b62b9ac89f8e0f337f246a244a62c84a\": rpc error: code = NotFound desc = could not find container \"1d1acf6805bd4e8073c12047295f2308b62b9ac89f8e0f337f246a244a62c84a\": container with ID starting with 1d1acf6805bd4e8073c12047295f2308b62b9ac89f8e0f337f246a244a62c84a not found: ID does not exist" Jul 29 08:56:16.603232 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.603210 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv"] Jul 29 08:56:16.607010 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:16.606987 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-pnfsv"] Jul 29 08:56:17.362480 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:17.362444 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15" path="/var/lib/kubelet/pods/e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15/volumes" Jul 29 08:56:35.675718 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.675687 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 08:56:35.676111 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.676021 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15" containerName="manager" Jul 29 08:56:35.676111 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.676032 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15" containerName="manager" Jul 29 08:56:35.676111 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.676052 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb519a7a-6863-4f9a-baa8-33b81f426a95" containerName="manager" Jul 29 08:56:35.676111 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.676058 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb519a7a-6863-4f9a-baa8-33b81f426a95" containerName="manager" Jul 29 08:56:35.676111 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.676106 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0b7f1f9-2d6b-463b-babe-fe4ec3a5af15" containerName="manager" Jul 29 08:56:35.676297 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.676116 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb519a7a-6863-4f9a-baa8-33b81f426a95" containerName="manager" Jul 29 08:56:35.679065 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.679050 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-824nb" Jul 29 08:56:35.681846 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.681802 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 08:56:35.683084 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.683062 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pcrnr\"" Jul 29 08:56:35.683084 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.683074 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 08:56:35.683198 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.683062 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 29 08:56:35.688205 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.688180 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 08:56:35.723229 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.723198 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 08:56:35.780623 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.780591 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/735bc652-2a81-41bd-aca9-bba5519283da-config-file\") pod \"limitador-limitador-54d6d988d5-824nb\" (UID: \"735bc652-2a81-41bd-aca9-bba5519283da\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-824nb" Jul 29 08:56:35.780771 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.780705 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2lx\" (UniqueName: \"kubernetes.io/projected/735bc652-2a81-41bd-aca9-bba5519283da-kube-api-access-ns2lx\") pod \"limitador-limitador-54d6d988d5-824nb\" (UID: \"735bc652-2a81-41bd-aca9-bba5519283da\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-824nb" Jul 29 08:56:35.881467 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.881427 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/735bc652-2a81-41bd-aca9-bba5519283da-config-file\") pod \"limitador-limitador-54d6d988d5-824nb\" (UID: \"735bc652-2a81-41bd-aca9-bba5519283da\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-824nb" Jul 29 08:56:35.881639 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.881516 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2lx\" (UniqueName: \"kubernetes.io/projected/735bc652-2a81-41bd-aca9-bba5519283da-kube-api-access-ns2lx\") pod \"limitador-limitador-54d6d988d5-824nb\" (UID: \"735bc652-2a81-41bd-aca9-bba5519283da\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-824nb" Jul 29 08:56:35.882070 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.882045 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/735bc652-2a81-41bd-aca9-bba5519283da-config-file\") pod \"limitador-limitador-54d6d988d5-824nb\" (UID: \"735bc652-2a81-41bd-aca9-bba5519283da\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-824nb" Jul 29 08:56:35.890199 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.890176 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2lx\" (UniqueName: \"kubernetes.io/projected/735bc652-2a81-41bd-aca9-bba5519283da-kube-api-access-ns2lx\") pod \"limitador-limitador-54d6d988d5-824nb\" (UID: \"735bc652-2a81-41bd-aca9-bba5519283da\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-824nb" Jul 29 08:56:35.990538 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:35.990468 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-824nb" Jul 29 08:56:36.119026 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:36.119004 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 08:56:36.121538 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:56:36.121498 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735bc652_2a81_41bd_aca9_bba5519283da.slice/crio-62f7fde4fd655458418c3df00e51b1ba43673df4109f865a0ce4eebecfb2f042 WatchSource:0}: Error finding container 62f7fde4fd655458418c3df00e51b1ba43673df4109f865a0ce4eebecfb2f042: Status 404 returned error can't find the container with id 62f7fde4fd655458418c3df00e51b1ba43673df4109f865a0ce4eebecfb2f042 Jul 29 08:56:36.346814 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:36.346725 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-xm84v"] Jul 29 08:56:36.351290 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:36.351268 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xm84v" Jul 29 08:56:36.354064 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:36.354043 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wtjt5\"" Jul 29 08:56:36.357465 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:36.357441 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xm84v"] Jul 29 08:56:36.487161 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:36.487130 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxhdq\" (UniqueName: \"kubernetes.io/projected/e99fd544-cb16-4b15-a380-0ec0b9d18f25-kube-api-access-hxhdq\") pod \"authorino-75d556bd77-xm84v\" (UID: \"e99fd544-cb16-4b15-a380-0ec0b9d18f25\") " pod="kuadrant-system/authorino-75d556bd77-xm84v" Jul 29 08:56:36.588441 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:36.588410 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxhdq\" (UniqueName: \"kubernetes.io/projected/e99fd544-cb16-4b15-a380-0ec0b9d18f25-kube-api-access-hxhdq\") pod \"authorino-75d556bd77-xm84v\" (UID: \"e99fd544-cb16-4b15-a380-0ec0b9d18f25\") " pod="kuadrant-system/authorino-75d556bd77-xm84v" Jul 29 08:56:36.598394 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:36.598306 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxhdq\" (UniqueName: \"kubernetes.io/projected/e99fd544-cb16-4b15-a380-0ec0b9d18f25-kube-api-access-hxhdq\") pod \"authorino-75d556bd77-xm84v\" (UID: \"e99fd544-cb16-4b15-a380-0ec0b9d18f25\") " pod="kuadrant-system/authorino-75d556bd77-xm84v" Jul 29 08:56:36.661591 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:36.661562 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xm84v" Jul 29 08:56:36.673913 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:36.673882 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-824nb" event={"ID":"735bc652-2a81-41bd-aca9-bba5519283da","Type":"ContainerStarted","Data":"62f7fde4fd655458418c3df00e51b1ba43673df4109f865a0ce4eebecfb2f042"} Jul 29 08:56:36.789709 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:36.789682 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xm84v"] Jul 29 08:56:36.791638 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:56:36.791610 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode99fd544_cb16_4b15_a380_0ec0b9d18f25.slice/crio-9b70938dcd7851c2cb5e7a028dda4e8da2d4a45afeb27f43c6aab13a3549a05c WatchSource:0}: Error finding container 9b70938dcd7851c2cb5e7a028dda4e8da2d4a45afeb27f43c6aab13a3549a05c: Status 404 returned error can't find the container with id 9b70938dcd7851c2cb5e7a028dda4e8da2d4a45afeb27f43c6aab13a3549a05c Jul 29 08:56:37.682098 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:37.682060 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xm84v" event={"ID":"e99fd544-cb16-4b15-a380-0ec0b9d18f25","Type":"ContainerStarted","Data":"9b70938dcd7851c2cb5e7a028dda4e8da2d4a45afeb27f43c6aab13a3549a05c"} Jul 29 08:56:41.708794 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:41.708740 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xm84v" event={"ID":"e99fd544-cb16-4b15-a380-0ec0b9d18f25","Type":"ContainerStarted","Data":"a6a6d605522b97a59476fae8d476b14c6435143714094e39557cb1d194e4112e"} Jul 29 08:56:41.710171 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:41.710144 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-824nb" event={"ID":"735bc652-2a81-41bd-aca9-bba5519283da","Type":"ContainerStarted","Data":"d6d1c99bf559829da186d73a7d02c0d2b14779e984c4ff272368a08c9231c927"} Jul 29 08:56:41.710311 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:41.710292 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-824nb" Jul 29 08:56:41.725519 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:41.725477 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-xm84v" podStartSLOduration=1.33232336 podStartE2EDuration="5.725462785s" totalImagesPullingTime="4.393139425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:56:41.723486983 +0000 UTC m=+514.847587782" watchObservedRunningTime="2026-07-29 08:56:41.725462785 +0000 UTC m=+514.849563585" Jul 29 08:56:52.717754 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:52.717718 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-824nb" Jul 29 08:56:52.746942 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:56:52.746894 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-824nb" podStartSLOduration=12.630562857 podStartE2EDuration="17.746877803s" totalImagesPullingTime="5.116314946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:56:41.742856759 +0000 UTC m=+514.866957562" watchObservedRunningTime="2026-07-29 08:56:52.746877803 +0000 UTC m=+525.870978603" Jul 29 08:57:11.537328 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:11.537292 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xm84v"] Jul 29 08:57:11.537827 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:11.537613 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-xm84v" podUID="e99fd544-cb16-4b15-a380-0ec0b9d18f25" containerName="authorino" containerID="cri-o://a6a6d605522b97a59476fae8d476b14c6435143714094e39557cb1d194e4112e" gracePeriod=30 Jul 29 08:57:11.785092 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:11.785064 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xm84v" Jul 29 08:57:11.785528 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:11.785507 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxhdq\" (UniqueName: \"kubernetes.io/projected/e99fd544-cb16-4b15-a380-0ec0b9d18f25-kube-api-access-hxhdq\") pod \"e99fd544-cb16-4b15-a380-0ec0b9d18f25\" (UID: \"e99fd544-cb16-4b15-a380-0ec0b9d18f25\") " Jul 29 08:57:11.787485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:11.787445 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99fd544-cb16-4b15-a380-0ec0b9d18f25-kube-api-access-hxhdq" (OuterVolumeSpecName: "kube-api-access-hxhdq") pod "e99fd544-cb16-4b15-a380-0ec0b9d18f25" (UID: "e99fd544-cb16-4b15-a380-0ec0b9d18f25"). InnerVolumeSpecName "kube-api-access-hxhdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 08:57:11.849373 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:11.849328 2563 generic.go:356] "Generic (PLEG): container finished" podID="e99fd544-cb16-4b15-a380-0ec0b9d18f25" containerID="a6a6d605522b97a59476fae8d476b14c6435143714094e39557cb1d194e4112e" exitCode=0 Jul 29 08:57:11.849527 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:11.849387 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xm84v" Jul 29 08:57:11.849527 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:11.849425 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xm84v" event={"ID":"e99fd544-cb16-4b15-a380-0ec0b9d18f25","Type":"ContainerDied","Data":"a6a6d605522b97a59476fae8d476b14c6435143714094e39557cb1d194e4112e"} Jul 29 08:57:11.849527 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:11.849470 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xm84v" event={"ID":"e99fd544-cb16-4b15-a380-0ec0b9d18f25","Type":"ContainerDied","Data":"9b70938dcd7851c2cb5e7a028dda4e8da2d4a45afeb27f43c6aab13a3549a05c"} Jul 29 08:57:11.849527 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:11.849494 2563 scope.go:117] "RemoveContainer" containerID="a6a6d605522b97a59476fae8d476b14c6435143714094e39557cb1d194e4112e" Jul 29 08:57:11.858418 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:11.858398 2563 scope.go:117] "RemoveContainer" containerID="a6a6d605522b97a59476fae8d476b14c6435143714094e39557cb1d194e4112e" Jul 29 08:57:11.858690 ip-10-0-141-66 kubenswrapper[2563]: E0729 08:57:11.858671 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6a6d605522b97a59476fae8d476b14c6435143714094e39557cb1d194e4112e\": container with ID starting with a6a6d605522b97a59476fae8d476b14c6435143714094e39557cb1d194e4112e not found: ID does not exist" containerID="a6a6d605522b97a59476fae8d476b14c6435143714094e39557cb1d194e4112e" Jul 29 08:57:11.858737 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:11.858699 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6a6d605522b97a59476fae8d476b14c6435143714094e39557cb1d194e4112e"} err="failed to get container status \"a6a6d605522b97a59476fae8d476b14c6435143714094e39557cb1d194e4112e\": rpc error: code = NotFound desc = could not find container \"a6a6d605522b97a59476fae8d476b14c6435143714094e39557cb1d194e4112e\": container with ID starting with a6a6d605522b97a59476fae8d476b14c6435143714094e39557cb1d194e4112e not found: ID does not exist" Jul 29 08:57:11.870544 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:11.870517 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xm84v"] Jul 29 08:57:11.873952 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:11.873928 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xm84v"] Jul 29 08:57:11.886693 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:11.886670 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxhdq\" (UniqueName: \"kubernetes.io/projected/e99fd544-cb16-4b15-a380-0ec0b9d18f25-kube-api-access-hxhdq\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 08:57:13.362347 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:13.362316 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e99fd544-cb16-4b15-a380-0ec0b9d18f25" path="/var/lib/kubelet/pods/e99fd544-cb16-4b15-a380-0ec0b9d18f25/volumes" Jul 29 08:57:55.167218 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:57:55.167183 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 08:58:30.918391 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:30.918287 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 08:58:38.958569 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:38.958527 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 08:58:53.014811 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.014775 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv"] Jul 29 08:58:53.015283 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.015265 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e99fd544-cb16-4b15-a380-0ec0b9d18f25" containerName="authorino" Jul 29 08:58:53.015328 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.015288 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e99fd544-cb16-4b15-a380-0ec0b9d18f25" containerName="authorino" Jul 29 08:58:53.015440 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.015427 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e99fd544-cb16-4b15-a380-0ec0b9d18f25" containerName="authorino" Jul 29 08:58:53.018857 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.018835 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.021542 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.021521 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 08:58:53.022804 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.022784 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-27jj4\"" Jul 29 08:58:53.022925 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.022790 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 08:58:53.022925 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.022906 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 29 08:58:53.028164 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.028142 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv"] Jul 29 08:58:53.056960 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.056931 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.057116 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.056980 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.057116 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.057005 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.057116 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.057029 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwjkg\" (UniqueName: \"kubernetes.io/projected/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-kube-api-access-hwjkg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.057116 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.057057 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.057116 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.057072 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.158444 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.158404 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.158444 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.158448 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwjkg\" (UniqueName: \"kubernetes.io/projected/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-kube-api-access-hwjkg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.158654 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.158478 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.158654 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.158505 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.158654 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.158555 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.158654 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.158626 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.158982 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.158953 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.158982 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.158982 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.159120 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.159047 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.160719 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.160701 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.160976 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.160947 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.166164 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.166140 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwjkg\" (UniqueName: \"kubernetes.io/projected/7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0-kube-api-access-hwjkg\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nb4jv\" (UID: \"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.331820 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.331731 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:58:53.459452 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.459426 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv"] Jul 29 08:58:53.461319 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:58:53.461286 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee02e5e_3a4a_4051_9ab9_a0218a0d8de0.slice/crio-ba81461c3ed2b213c8503bacde7d06737a3bd8f95ca873a6cd656419c3b7f8c5 WatchSource:0}: Error finding container ba81461c3ed2b213c8503bacde7d06737a3bd8f95ca873a6cd656419c3b7f8c5: Status 404 returned error can't find the container with id ba81461c3ed2b213c8503bacde7d06737a3bd8f95ca873a6cd656419c3b7f8c5 Jul 29 08:58:53.462628 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:53.462610 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 08:58:54.062485 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:54.062450 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 08:58:54.310072 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:54.310026 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" event={"ID":"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0","Type":"ContainerStarted","Data":"ba81461c3ed2b213c8503bacde7d06737a3bd8f95ca873a6cd656419c3b7f8c5"} Jul 29 08:58:55.315958 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:55.315923 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" event={"ID":"7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0","Type":"ContainerStarted","Data":"16c2c084c4c047b1f4fd9c98b451598328c92d8fad2f965ed7db08609fef75da"} Jul 29 08:58:55.334228 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:58:55.334180 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" podStartSLOduration=2.301899407 podStartE2EDuration="3.334162788s" totalImagesPullingTime="1.032263381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:58:55.332525849 +0000 UTC m=+648.456626673" watchObservedRunningTime="2026-07-29 08:58:55.334162788 +0000 UTC m=+648.458263564" Jul 29 08:59:02.118516 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.118476 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l"] Jul 29 08:59:02.121923 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.121901 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.124588 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.124568 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 29 08:59:02.132890 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.132869 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l"] Jul 29 08:59:02.245498 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.245464 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.245498 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.245498 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.245751 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.245540 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.245751 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.245643 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.245751 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.245690 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcb2p\" (UniqueName: \"kubernetes.io/projected/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-kube-api-access-lcb2p\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.245751 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.245732 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.346761 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.346721 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.346938 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.346767 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcb2p\" (UniqueName: \"kubernetes.io/projected/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-kube-api-access-lcb2p\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.346938 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.346794 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.346938 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.346861 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.346938 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.346883 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.346938 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.346925 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.347235 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.347206 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.347350 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.347258 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.347350 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.347315 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.349146 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.349128 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.349293 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.349275 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.354587 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.354569 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcb2p\" (UniqueName: \"kubernetes.io/projected/6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6-kube-api-access-lcb2p\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l\" (UID: \"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.432404 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.432376 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:02.564146 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:02.564075 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l"] Jul 29 08:59:02.566572 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:59:02.566547 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6793c0d0_ee0a_42dc_a23a_12a9a7afdcb6.slice/crio-a8bc5c39e9d5060a842cbbb3c2a4bf40e9dc98b46cacd235c85eddd537b24eea WatchSource:0}: Error finding container a8bc5c39e9d5060a842cbbb3c2a4bf40e9dc98b46cacd235c85eddd537b24eea: Status 404 returned error can't find the container with id a8bc5c39e9d5060a842cbbb3c2a4bf40e9dc98b46cacd235c85eddd537b24eea Jul 29 08:59:03.332400 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:03.332346 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:59:03.332784 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:03.332434 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:59:03.344746 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:03.344718 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:59:03.362860 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:03.362826 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" event={"ID":"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6","Type":"ContainerStarted","Data":"a8bc5c39e9d5060a842cbbb3c2a4bf40e9dc98b46cacd235c85eddd537b24eea"} Jul 29 08:59:03.372571 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:03.372553 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nb4jv" Jul 29 08:59:03.567779 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:03.567745 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 08:59:06.376924 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:06.376879 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" event={"ID":"6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6","Type":"ContainerStarted","Data":"96e0f4b3d6528a387591381978a90aab91a17a1bc4a6497f4027f8528dd50f1b"} Jul 29 08:59:06.396700 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:06.396647 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" podStartSLOduration=1.618877596 podStartE2EDuration="4.39663052s" totalImagesPullingTime="2.777752924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:59:06.394866511 +0000 UTC m=+659.518967308" watchObservedRunningTime="2026-07-29 08:59:06.39663052 +0000 UTC m=+659.520731320" Jul 29 08:59:12.433383 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:12.433335 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:12.433780 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:12.433416 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:12.445667 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:12.445645 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:13.420449 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:13.420420 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l" Jul 29 08:59:13.556721 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:13.556686 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 08:59:25.815483 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:25.815446 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z"] Jul 29 08:59:25.822053 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:25.822024 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:25.825913 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:25.825889 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 29 08:59:25.832723 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:25.832698 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z"] Jul 29 08:59:25.955058 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:25.955022 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66257daf-2ac4-48cc-8b39-aec37967a92a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:25.955235 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:25.955066 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66257daf-2ac4-48cc-8b39-aec37967a92a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:25.955235 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:25.955098 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66257daf-2ac4-48cc-8b39-aec37967a92a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:25.955235 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:25.955151 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lp9r\" (UniqueName: \"kubernetes.io/projected/66257daf-2ac4-48cc-8b39-aec37967a92a-kube-api-access-4lp9r\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:25.955235 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:25.955195 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66257daf-2ac4-48cc-8b39-aec37967a92a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:25.955433 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:25.955245 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66257daf-2ac4-48cc-8b39-aec37967a92a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:26.056123 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:26.056080 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66257daf-2ac4-48cc-8b39-aec37967a92a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:26.056328 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:26.056138 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66257daf-2ac4-48cc-8b39-aec37967a92a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:26.056328 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:26.056170 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lp9r\" (UniqueName: \"kubernetes.io/projected/66257daf-2ac4-48cc-8b39-aec37967a92a-kube-api-access-4lp9r\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:26.056328 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:26.056211 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66257daf-2ac4-48cc-8b39-aec37967a92a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:26.056328 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:26.056271 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66257daf-2ac4-48cc-8b39-aec37967a92a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:26.056328 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:26.056310 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66257daf-2ac4-48cc-8b39-aec37967a92a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:26.056584 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:26.056514 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66257daf-2ac4-48cc-8b39-aec37967a92a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:26.056584 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:26.056559 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66257daf-2ac4-48cc-8b39-aec37967a92a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:26.056690 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:26.056672 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66257daf-2ac4-48cc-8b39-aec37967a92a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:26.058567 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:26.058543 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66257daf-2ac4-48cc-8b39-aec37967a92a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:26.058727 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:26.058710 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66257daf-2ac4-48cc-8b39-aec37967a92a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:26.064855 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:26.064828 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lp9r\" (UniqueName: \"kubernetes.io/projected/66257daf-2ac4-48cc-8b39-aec37967a92a-kube-api-access-4lp9r\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z\" (UID: \"66257daf-2ac4-48cc-8b39-aec37967a92a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:26.133832 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:26.133752 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:26.266436 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:26.266409 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z"] Jul 29 08:59:26.268776 ip-10-0-141-66 kubenswrapper[2563]: W0729 08:59:26.268744 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66257daf_2ac4_48cc_8b39_aec37967a92a.slice/crio-ffd869ceb4a2d14052e7f6db179a76d0a6b484af07f80830e78ff9276e0b6a54 WatchSource:0}: Error finding container ffd869ceb4a2d14052e7f6db179a76d0a6b484af07f80830e78ff9276e0b6a54: Status 404 returned error can't find the container with id ffd869ceb4a2d14052e7f6db179a76d0a6b484af07f80830e78ff9276e0b6a54 Jul 29 08:59:26.468628 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:26.468593 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" event={"ID":"66257daf-2ac4-48cc-8b39-aec37967a92a","Type":"ContainerStarted","Data":"ffd869ceb4a2d14052e7f6db179a76d0a6b484af07f80830e78ff9276e0b6a54"} Jul 29 08:59:27.474337 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:27.474305 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" event={"ID":"66257daf-2ac4-48cc-8b39-aec37967a92a","Type":"ContainerStarted","Data":"5d41a70dd2ef2fb3b243b71dc4facb8b43bf89d829cb2dce5d24e2297d5d4b36"} Jul 29 08:59:27.492963 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:27.492914 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" podStartSLOduration=2.119136161 podStartE2EDuration="2.4928992s" totalImagesPullingTime="373.763039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 08:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 08:59:27.491137357 +0000 UTC m=+680.615238156" watchObservedRunningTime="2026-07-29 08:59:27.4928992 +0000 UTC m=+680.617000001" Jul 29 08:59:28.373161 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:28.373123 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 08:59:36.134533 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:36.134494 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:36.135056 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:36.134546 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:36.147074 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:36.147051 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 08:59:36.524567 ip-10-0-141-66 kubenswrapper[2563]: I0729 08:59:36.524535 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z" Jul 29 09:00:34.617214 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:34.617182 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b649d98dc-k4wnc"] Jul 29 09:00:34.620434 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:34.620418 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b649d98dc-k4wnc" Jul 29 09:00:34.623072 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:34.623045 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wtjt5\"" Jul 29 09:00:34.624183 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:34.624166 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 29 09:00:34.628276 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:34.628253 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b649d98dc-k4wnc"] Jul 29 09:00:34.733003 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:34.732969 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d1c241f4-c19e-4ab8-987a-e8abfce03ca6-tls-cert\") pod \"authorino-7b649d98dc-k4wnc\" (UID: \"d1c241f4-c19e-4ab8-987a-e8abfce03ca6\") " pod="kuadrant-system/authorino-7b649d98dc-k4wnc" Jul 29 09:00:34.733167 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:34.733010 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg9cr\" (UniqueName: \"kubernetes.io/projected/d1c241f4-c19e-4ab8-987a-e8abfce03ca6-kube-api-access-lg9cr\") pod \"authorino-7b649d98dc-k4wnc\" (UID: \"d1c241f4-c19e-4ab8-987a-e8abfce03ca6\") " pod="kuadrant-system/authorino-7b649d98dc-k4wnc" Jul 29 09:00:34.833540 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:34.833507 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d1c241f4-c19e-4ab8-987a-e8abfce03ca6-tls-cert\") pod \"authorino-7b649d98dc-k4wnc\" (UID: \"d1c241f4-c19e-4ab8-987a-e8abfce03ca6\") " pod="kuadrant-system/authorino-7b649d98dc-k4wnc" Jul 29 09:00:34.833540 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:34.833552 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg9cr\" (UniqueName: \"kubernetes.io/projected/d1c241f4-c19e-4ab8-987a-e8abfce03ca6-kube-api-access-lg9cr\") pod \"authorino-7b649d98dc-k4wnc\" (UID: \"d1c241f4-c19e-4ab8-987a-e8abfce03ca6\") " pod="kuadrant-system/authorino-7b649d98dc-k4wnc" Jul 29 09:00:34.835903 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:34.835873 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d1c241f4-c19e-4ab8-987a-e8abfce03ca6-tls-cert\") pod \"authorino-7b649d98dc-k4wnc\" (UID: \"d1c241f4-c19e-4ab8-987a-e8abfce03ca6\") " pod="kuadrant-system/authorino-7b649d98dc-k4wnc" Jul 29 09:00:34.841432 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:34.841410 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg9cr\" (UniqueName: \"kubernetes.io/projected/d1c241f4-c19e-4ab8-987a-e8abfce03ca6-kube-api-access-lg9cr\") pod \"authorino-7b649d98dc-k4wnc\" (UID: \"d1c241f4-c19e-4ab8-987a-e8abfce03ca6\") " pod="kuadrant-system/authorino-7b649d98dc-k4wnc" Jul 29 09:00:34.930176 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:34.930154 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b649d98dc-k4wnc" Jul 29 09:00:35.063973 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:35.063950 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b649d98dc-k4wnc"] Jul 29 09:00:35.066067 ip-10-0-141-66 kubenswrapper[2563]: W0729 09:00:35.066041 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1c241f4_c19e_4ab8_987a_e8abfce03ca6.slice/crio-b32fa9964815bc137a0bbfcbea02ba171608a4f1d3fc066eb119ebe1dfc1078b WatchSource:0}: Error finding container b32fa9964815bc137a0bbfcbea02ba171608a4f1d3fc066eb119ebe1dfc1078b: Status 404 returned error can't find the container with id b32fa9964815bc137a0bbfcbea02ba171608a4f1d3fc066eb119ebe1dfc1078b Jul 29 09:00:35.790395 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:35.790272 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b649d98dc-k4wnc" event={"ID":"d1c241f4-c19e-4ab8-987a-e8abfce03ca6","Type":"ContainerStarted","Data":"237addcbe43566af3970d963e920652ba05fb055cb0c2940cf38fb158e35743e"} Jul 29 09:00:35.790395 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:35.790317 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b649d98dc-k4wnc" event={"ID":"d1c241f4-c19e-4ab8-987a-e8abfce03ca6","Type":"ContainerStarted","Data":"b32fa9964815bc137a0bbfcbea02ba171608a4f1d3fc066eb119ebe1dfc1078b"} Jul 29 09:00:35.810037 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:00:35.809787 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b649d98dc-k4wnc" podStartSLOduration=1.343083645 podStartE2EDuration="1.809765305s" totalImagesPullingTime="466.68166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 09:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 09:00:35.80931522 +0000 UTC m=+748.933416011" watchObservedRunningTime="2026-07-29 09:00:35.809765305 +0000 UTC m=+748.933866100" Jul 29 09:01:05.268238 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:01:05.268206 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:01:14.479470 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:01:14.479434 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:01:25.261850 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:01:25.261813 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:03:04.738351 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:03:04.738276 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:03:15.660131 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:03:15.660094 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:03:22.664128 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:03:22.664088 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:03:34.961496 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:03:34.961458 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:03:52.756785 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:03:52.756747 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:04:01.754863 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:04:01.754826 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:04:11.857988 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:04:11.857952 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:04:21.056868 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:04:21.056832 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:04:31.466904 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:04:31.466820 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:04:40.862855 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:04:40.862822 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:04:52.557916 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:04:52.557881 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:05:02.157676 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:05:02.157645 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:05:24.862239 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:05:24.862202 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:05:36.967979 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:05:36.967944 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:06:00.358320 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:06:00.358234 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:06:12.755150 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:06:12.755102 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:06:21.962107 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:06:21.962069 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:06:39.560429 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:06:39.560390 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:06:48.373220 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:06:48.373182 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:07:01.555015 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:07:01.554979 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:07:22.404078 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:07:22.404020 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:07:32.663985 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:07:32.663885 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:09:17.130204 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:09:17.130168 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:09:28.822045 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:09:28.822010 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:09:45.721426 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:09:45.721388 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:09:56.016780 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:09:56.016737 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:10:07.015629 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:10:07.015591 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:10:16.418625 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:10:16.418591 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:10:27.012572 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:10:27.012536 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:10:36.115614 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:10:36.115524 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:10:48.821548 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:10:48.821512 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:10:56.918847 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:10:56.918811 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:11:07.532731 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:07.532697 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:11:32.010453 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:32.010414 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:11:39.622115 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:39.622076 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:11:51.733519 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:51.733487 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:11:53.722875 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.722839 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk"] Jul 29 09:11:53.726871 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.726846 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.729824 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.729801 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-3dafdacb-kserve-self-signed-certs\"" Jul 29 09:11:53.738033 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.738004 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk"] Jul 29 09:11:53.818134 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.818090 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-home\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.818331 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.818149 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-model-cache\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.818331 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.818215 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvnf4\" (UniqueName: \"kubernetes.io/projected/d99fe400-cc38-4655-ab95-0c0a816cd610-kube-api-access-rvnf4\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.818331 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.818285 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-dshm\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.818331 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.818304 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99fe400-cc38-4655-ab95-0c0a816cd610-tls-certs\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.818603 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.818340 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-tmp-dir\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.919429 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.919390 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-model-cache\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.919429 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.919426 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvnf4\" (UniqueName: \"kubernetes.io/projected/d99fe400-cc38-4655-ab95-0c0a816cd610-kube-api-access-rvnf4\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.919627 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.919482 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-dshm\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.919627 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.919515 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99fe400-cc38-4655-ab95-0c0a816cd610-tls-certs\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.919627 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.919552 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-tmp-dir\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.919627 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.919590 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-home\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.920029 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.920003 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-model-cache\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.920109 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.920046 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-home\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.920109 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.920072 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-tmp-dir\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.921991 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.921959 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-dshm\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.922309 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.922286 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99fe400-cc38-4655-ab95-0c0a816cd610-tls-certs\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:53.927380 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:53.927339 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvnf4\" (UniqueName: \"kubernetes.io/projected/d99fe400-cc38-4655-ab95-0c0a816cd610-kube-api-access-rvnf4\") pod \"e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:54.037416 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:54.037318 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:11:54.204965 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:54.204938 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk"] Jul 29 09:11:54.207173 ip-10-0-141-66 kubenswrapper[2563]: W0729 09:11:54.207127 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd99fe400_cc38_4655_ab95_0c0a816cd610.slice/crio-ecb5bda15d3ac3dd3b8e0c92faae7e0d296d876577baa70b7472db4e38d0541a WatchSource:0}: Error finding container ecb5bda15d3ac3dd3b8e0c92faae7e0d296d876577baa70b7472db4e38d0541a: Status 404 returned error can't find the container with id ecb5bda15d3ac3dd3b8e0c92faae7e0d296d876577baa70b7472db4e38d0541a Jul 29 09:11:55.008657 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:55.008620 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" event={"ID":"d99fe400-cc38-4655-ab95-0c0a816cd610","Type":"ContainerStarted","Data":"89345ebe1a2c2d20a28b2fc6a5ab00d26593823558fc56ad306c433dce27365d"} Jul 29 09:11:55.008657 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:55.008662 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" event={"ID":"d99fe400-cc38-4655-ab95-0c0a816cd610","Type":"ContainerStarted","Data":"ecb5bda15d3ac3dd3b8e0c92faae7e0d296d876577baa70b7472db4e38d0541a"} Jul 29 09:11:55.026322 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:11:55.026270 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" podStartSLOduration=2.026251909 podStartE2EDuration="2.026251909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 09:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 09:11:55.025447951 +0000 UTC m=+1428.149548751" watchObservedRunningTime="2026-07-29 09:11:55.026251909 +0000 UTC m=+1428.150352708" Jul 29 09:12:04.037568 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:04.037486 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:12:04.037971 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:04.037765 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:12:04.050879 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:04.050852 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:12:05.064809 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:05.064772 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:12:05.819281 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:05.819241 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:12:27.960904 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:27.960868 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk"] Jul 29 09:12:30.308741 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:30.308704 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:12:42.982317 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:42.982246 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" podUID="d99fe400-cc38-4655-ab95-0c0a816cd610" containerName="main" containerID="cri-o://89345ebe1a2c2d20a28b2fc6a5ab00d26593823558fc56ad306c433dce27365d" gracePeriod=45 Jul 29 09:12:43.229703 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.229679 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:12:43.238458 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.238394 2563 generic.go:356] "Generic (PLEG): container finished" podID="d99fe400-cc38-4655-ab95-0c0a816cd610" containerID="89345ebe1a2c2d20a28b2fc6a5ab00d26593823558fc56ad306c433dce27365d" exitCode=0 Jul 29 09:12:43.238458 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.238441 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" event={"ID":"d99fe400-cc38-4655-ab95-0c0a816cd610","Type":"ContainerDied","Data":"89345ebe1a2c2d20a28b2fc6a5ab00d26593823558fc56ad306c433dce27365d"} Jul 29 09:12:43.238588 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.238473 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" event={"ID":"d99fe400-cc38-4655-ab95-0c0a816cd610","Type":"ContainerDied","Data":"ecb5bda15d3ac3dd3b8e0c92faae7e0d296d876577baa70b7472db4e38d0541a"} Jul 29 09:12:43.238588 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.238477 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk" Jul 29 09:12:43.238588 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.238490 2563 scope.go:117] "RemoveContainer" containerID="89345ebe1a2c2d20a28b2fc6a5ab00d26593823558fc56ad306c433dce27365d" Jul 29 09:12:43.247314 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.247281 2563 scope.go:117] "RemoveContainer" containerID="89345ebe1a2c2d20a28b2fc6a5ab00d26593823558fc56ad306c433dce27365d" Jul 29 09:12:43.247961 ip-10-0-141-66 kubenswrapper[2563]: E0729 09:12:43.247919 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89345ebe1a2c2d20a28b2fc6a5ab00d26593823558fc56ad306c433dce27365d\": container with ID starting with 89345ebe1a2c2d20a28b2fc6a5ab00d26593823558fc56ad306c433dce27365d not found: ID does not exist" containerID="89345ebe1a2c2d20a28b2fc6a5ab00d26593823558fc56ad306c433dce27365d" Jul 29 09:12:43.248063 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.247959 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89345ebe1a2c2d20a28b2fc6a5ab00d26593823558fc56ad306c433dce27365d"} err="failed to get container status \"89345ebe1a2c2d20a28b2fc6a5ab00d26593823558fc56ad306c433dce27365d\": rpc error: code = NotFound desc = could not find container \"89345ebe1a2c2d20a28b2fc6a5ab00d26593823558fc56ad306c433dce27365d\": container with ID starting with 89345ebe1a2c2d20a28b2fc6a5ab00d26593823558fc56ad306c433dce27365d not found: ID does not exist" Jul 29 09:12:43.314335 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.314305 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:12:43.354716 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.354686 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-dshm\") pod \"d99fe400-cc38-4655-ab95-0c0a816cd610\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " Jul 29 09:12:43.354716 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.354719 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-model-cache\") pod \"d99fe400-cc38-4655-ab95-0c0a816cd610\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " Jul 29 09:12:43.354941 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.354765 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99fe400-cc38-4655-ab95-0c0a816cd610-tls-certs\") pod \"d99fe400-cc38-4655-ab95-0c0a816cd610\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " Jul 29 09:12:43.354941 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.354806 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-home\") pod \"d99fe400-cc38-4655-ab95-0c0a816cd610\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " Jul 29 09:12:43.354941 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.354827 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvnf4\" (UniqueName: \"kubernetes.io/projected/d99fe400-cc38-4655-ab95-0c0a816cd610-kube-api-access-rvnf4\") pod \"d99fe400-cc38-4655-ab95-0c0a816cd610\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " Jul 29 09:12:43.354941 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.354857 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-tmp-dir\") pod \"d99fe400-cc38-4655-ab95-0c0a816cd610\" (UID: \"d99fe400-cc38-4655-ab95-0c0a816cd610\") " Jul 29 09:12:43.355201 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.355011 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-model-cache" (OuterVolumeSpecName: "model-cache") pod "d99fe400-cc38-4655-ab95-0c0a816cd610" (UID: "d99fe400-cc38-4655-ab95-0c0a816cd610"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:12:43.355201 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.355074 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-home" (OuterVolumeSpecName: "home") pod "d99fe400-cc38-4655-ab95-0c0a816cd610" (UID: "d99fe400-cc38-4655-ab95-0c0a816cd610"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:12:43.355201 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.355168 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d99fe400-cc38-4655-ab95-0c0a816cd610" (UID: "d99fe400-cc38-4655-ab95-0c0a816cd610"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:12:43.355201 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.355200 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:12:43.355384 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.355212 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:12:43.357129 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.357097 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-dshm" (OuterVolumeSpecName: "dshm") pod "d99fe400-cc38-4655-ab95-0c0a816cd610" (UID: "d99fe400-cc38-4655-ab95-0c0a816cd610"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:12:43.357228 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.357167 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d99fe400-cc38-4655-ab95-0c0a816cd610-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d99fe400-cc38-4655-ab95-0c0a816cd610" (UID: "d99fe400-cc38-4655-ab95-0c0a816cd610"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 09:12:43.357228 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.357202 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99fe400-cc38-4655-ab95-0c0a816cd610-kube-api-access-rvnf4" (OuterVolumeSpecName: "kube-api-access-rvnf4") pod "d99fe400-cc38-4655-ab95-0c0a816cd610" (UID: "d99fe400-cc38-4655-ab95-0c0a816cd610"). InnerVolumeSpecName "kube-api-access-rvnf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 09:12:43.455724 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.455688 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvnf4\" (UniqueName: \"kubernetes.io/projected/d99fe400-cc38-4655-ab95-0c0a816cd610-kube-api-access-rvnf4\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:12:43.455724 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.455723 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:12:43.455986 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.455738 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d99fe400-cc38-4655-ab95-0c0a816cd610-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:12:43.455986 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.455750 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99fe400-cc38-4655-ab95-0c0a816cd610-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:12:43.554704 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.554629 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk"] Jul 29 09:12:43.558774 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:43.558752 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-3dafdacb-kserve-56c68f69c5-qv4bk"] Jul 29 09:12:45.362123 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:45.362088 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d99fe400-cc38-4655-ab95-0c0a816cd610" path="/var/lib/kubelet/pods/d99fe400-cc38-4655-ab95-0c0a816cd610/volumes" Jul 29 09:12:53.722481 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:12:53.722438 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:13:06.438599 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:13:06.438561 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:13:16.023021 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:13:16.022985 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:13:25.320788 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:13:25.320751 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:13:33.323970 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:13:33.323873 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:13:41.520016 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:13:41.519977 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:13:53.222068 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:13:53.222033 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:14:06.715876 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:14:06.715838 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:14:15.925618 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:14:15.925580 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:14:25.527069 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:14:25.527038 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:14:38.016142 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:14:38.016105 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:14:50.521371 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:14:50.521323 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:14:58.616635 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:14:58.616597 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:15:11.015317 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:15:11.015268 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:15:23.621082 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:15:23.621041 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:15:33.118189 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:15:33.118147 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:15:42.909150 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:15:42.909111 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:15:56.845565 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:15:56.845528 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:16:08.427489 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:16:08.427456 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:16:24.017147 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:16:24.017109 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:16:50.762393 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:16:50.762288 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:17:01.115520 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:17:01.115488 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:17:26.120223 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:17:26.120186 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:17:36.214172 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:17:36.214136 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:17:58.117566 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:17:58.117525 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:18:08.617276 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:18:08.617241 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:18:35.414895 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:18:35.414855 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:18:46.517147 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:18:46.517110 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:19:57.017011 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:19:57.016917 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:20:02.941543 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:20:02.941507 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:20:13.717910 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:20:13.717869 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:20:26.822945 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:20:26.822903 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:21:03.117865 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:21:03.117789 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:21:36.715687 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:21:36.715649 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:22:02.314987 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:22:02.314945 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:22:17.026293 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:22:17.026254 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:22:38.422870 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:22:38.422782 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:22:48.826440 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:22:48.826401 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:23:49.237423 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:23:49.237390 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:24:01.523511 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:24:01.523430 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:24:51.446269 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:24:51.446229 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:25:20.334388 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:20.334343 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:25:42.726804 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.726721 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx"] Jul 29 09:25:42.727228 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.727140 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d99fe400-cc38-4655-ab95-0c0a816cd610" containerName="main" Jul 29 09:25:42.727228 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.727155 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d99fe400-cc38-4655-ab95-0c0a816cd610" containerName="main" Jul 29 09:25:42.727301 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.727246 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d99fe400-cc38-4655-ab95-0c0a816cd610" containerName="main" Jul 29 09:25:42.730109 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.730093 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.733287 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.733264 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-306d65\"/\"openshift-service-ca.crt\"" Jul 29 09:25:42.734525 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.734505 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-306d65\"/\"default-dockercfg-fclxd\"" Jul 29 09:25:42.734628 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.734549 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-306d65\"/\"kube-root-ca.crt\"" Jul 29 09:25:42.734741 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.734727 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-306d65\"/\"test-model-306d65-kserve-self-signed-certs\"" Jul 29 09:25:42.746194 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.746171 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx"] Jul 29 09:25:42.828623 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.828589 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znk99\" (UniqueName: \"kubernetes.io/projected/0d904ef5-2210-48f3-8792-002854ed66b1-kube-api-access-znk99\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.828776 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.828629 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-dshm\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.828776 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.828648 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-tmp-dir\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.828776 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.828711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d904ef5-2210-48f3-8792-002854ed66b1-tls-certs\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.828776 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.828753 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-model-cache\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.828913 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.828784 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-home\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.929330 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.929300 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znk99\" (UniqueName: \"kubernetes.io/projected/0d904ef5-2210-48f3-8792-002854ed66b1-kube-api-access-znk99\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.929513 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.929342 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-dshm\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.929513 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.929379 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-tmp-dir\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.929513 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.929409 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d904ef5-2210-48f3-8792-002854ed66b1-tls-certs\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.929513 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.929440 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-model-cache\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.929513 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.929484 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-home\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.929890 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.929859 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-tmp-dir\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.930013 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.929888 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-model-cache\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.930013 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.929931 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-home\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.931621 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.931589 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-dshm\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.931864 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.931845 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d904ef5-2210-48f3-8792-002854ed66b1-tls-certs\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:42.948752 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:42.948728 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znk99\" (UniqueName: \"kubernetes.io/projected/0d904ef5-2210-48f3-8792-002854ed66b1-kube-api-access-znk99\") pod \"test-model-306d65-kserve-7dffffcd85-p2tdx\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:43.038919 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:43.038832 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:43.174398 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:43.174354 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx"] Jul 29 09:25:43.176073 ip-10-0-141-66 kubenswrapper[2563]: W0729 09:25:43.176046 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d904ef5_2210_48f3_8792_002854ed66b1.slice/crio-5c88208bfe623fda1c9f24361877625bf1c1138e32de29843a3a584a34205f0f WatchSource:0}: Error finding container 5c88208bfe623fda1c9f24361877625bf1c1138e32de29843a3a584a34205f0f: Status 404 returned error can't find the container with id 5c88208bfe623fda1c9f24361877625bf1c1138e32de29843a3a584a34205f0f Jul 29 09:25:43.926967 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:43.926930 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" event={"ID":"0d904ef5-2210-48f3-8792-002854ed66b1","Type":"ContainerStarted","Data":"b284f7f15e12f58212aea896482f7f499bb735e67b80936c011ef81a20c8385c"} Jul 29 09:25:43.926967 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:43.926974 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" event={"ID":"0d904ef5-2210-48f3-8792-002854ed66b1","Type":"ContainerStarted","Data":"5c88208bfe623fda1c9f24361877625bf1c1138e32de29843a3a584a34205f0f"} Jul 29 09:25:43.945699 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:43.945645 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" podStartSLOduration=1.9456281560000002 podStartE2EDuration="1.945628156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 09:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 09:25:43.945125423 +0000 UTC m=+2257.069226222" watchObservedRunningTime="2026-07-29 09:25:43.945628156 +0000 UTC m=+2257.069728966" Jul 29 09:25:47.221844 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:47.221812 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:25:53.039566 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:53.039532 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:53.039966 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:53.039602 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:53.052212 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:53.052183 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:25:53.984628 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:25:53.984600 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:26:15.426429 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:26:15.426390 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:26:29.828388 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:26:29.828322 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:26:52.841975 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:26:52.841937 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx"] Jul 29 09:27:00.325385 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:00.325288 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:27:07.861587 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:07.861523 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" podUID="0d904ef5-2210-48f3-8792-002854ed66b1" containerName="main" containerID="cri-o://b284f7f15e12f58212aea896482f7f499bb735e67b80936c011ef81a20c8385c" gracePeriod=45 Jul 29 09:27:08.103633 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.103604 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:27:08.201312 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.201283 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d904ef5-2210-48f3-8792-002854ed66b1-tls-certs\") pod \"0d904ef5-2210-48f3-8792-002854ed66b1\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " Jul 29 09:27:08.201312 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.201320 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-tmp-dir\") pod \"0d904ef5-2210-48f3-8792-002854ed66b1\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " Jul 29 09:27:08.201525 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.201358 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-model-cache\") pod \"0d904ef5-2210-48f3-8792-002854ed66b1\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " Jul 29 09:27:08.201525 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.201408 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-home\") pod \"0d904ef5-2210-48f3-8792-002854ed66b1\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " Jul 29 09:27:08.201525 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.201445 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znk99\" (UniqueName: \"kubernetes.io/projected/0d904ef5-2210-48f3-8792-002854ed66b1-kube-api-access-znk99\") pod \"0d904ef5-2210-48f3-8792-002854ed66b1\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " Jul 29 09:27:08.201525 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.201503 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-dshm\") pod \"0d904ef5-2210-48f3-8792-002854ed66b1\" (UID: \"0d904ef5-2210-48f3-8792-002854ed66b1\") " Jul 29 09:27:08.201730 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.201622 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d904ef5-2210-48f3-8792-002854ed66b1" (UID: "0d904ef5-2210-48f3-8792-002854ed66b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:27:08.201730 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.201636 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-home" (OuterVolumeSpecName: "home") pod "0d904ef5-2210-48f3-8792-002854ed66b1" (UID: "0d904ef5-2210-48f3-8792-002854ed66b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:27:08.201730 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.201661 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d904ef5-2210-48f3-8792-002854ed66b1" (UID: "0d904ef5-2210-48f3-8792-002854ed66b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:27:08.201882 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.201801 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:27:08.201882 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.201813 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:27:08.201882 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.201822 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:27:08.203556 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.203525 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d904ef5-2210-48f3-8792-002854ed66b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d904ef5-2210-48f3-8792-002854ed66b1" (UID: "0d904ef5-2210-48f3-8792-002854ed66b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 09:27:08.203670 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.203624 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-dshm" (OuterVolumeSpecName: "dshm") pod "0d904ef5-2210-48f3-8792-002854ed66b1" (UID: "0d904ef5-2210-48f3-8792-002854ed66b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:27:08.203670 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.203629 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d904ef5-2210-48f3-8792-002854ed66b1-kube-api-access-znk99" (OuterVolumeSpecName: "kube-api-access-znk99") pod "0d904ef5-2210-48f3-8792-002854ed66b1" (UID: "0d904ef5-2210-48f3-8792-002854ed66b1"). InnerVolumeSpecName "kube-api-access-znk99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 09:27:08.305389 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.303440 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d904ef5-2210-48f3-8792-002854ed66b1-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:27:08.305389 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.303479 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znk99\" (UniqueName: \"kubernetes.io/projected/0d904ef5-2210-48f3-8792-002854ed66b1-kube-api-access-znk99\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:27:08.305389 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.303494 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d904ef5-2210-48f3-8792-002854ed66b1-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:27:08.332290 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.332253 2563 generic.go:356] "Generic (PLEG): container finished" podID="0d904ef5-2210-48f3-8792-002854ed66b1" containerID="b284f7f15e12f58212aea896482f7f499bb735e67b80936c011ef81a20c8385c" exitCode=0 Jul 29 09:27:08.332476 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.332315 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" Jul 29 09:27:08.332476 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.332348 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" event={"ID":"0d904ef5-2210-48f3-8792-002854ed66b1","Type":"ContainerDied","Data":"b284f7f15e12f58212aea896482f7f499bb735e67b80936c011ef81a20c8385c"} Jul 29 09:27:08.332476 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.332413 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx" event={"ID":"0d904ef5-2210-48f3-8792-002854ed66b1","Type":"ContainerDied","Data":"5c88208bfe623fda1c9f24361877625bf1c1138e32de29843a3a584a34205f0f"} Jul 29 09:27:08.332476 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.332432 2563 scope.go:117] "RemoveContainer" containerID="b284f7f15e12f58212aea896482f7f499bb735e67b80936c011ef81a20c8385c" Jul 29 09:27:08.341578 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.341561 2563 scope.go:117] "RemoveContainer" containerID="b284f7f15e12f58212aea896482f7f499bb735e67b80936c011ef81a20c8385c" Jul 29 09:27:08.341835 ip-10-0-141-66 kubenswrapper[2563]: E0729 09:27:08.341815 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b284f7f15e12f58212aea896482f7f499bb735e67b80936c011ef81a20c8385c\": container with ID starting with b284f7f15e12f58212aea896482f7f499bb735e67b80936c011ef81a20c8385c not found: ID does not exist" containerID="b284f7f15e12f58212aea896482f7f499bb735e67b80936c011ef81a20c8385c" Jul 29 09:27:08.341886 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.341846 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b284f7f15e12f58212aea896482f7f499bb735e67b80936c011ef81a20c8385c"} err="failed to get container status \"b284f7f15e12f58212aea896482f7f499bb735e67b80936c011ef81a20c8385c\": rpc error: code = NotFound desc = could not find container \"b284f7f15e12f58212aea896482f7f499bb735e67b80936c011ef81a20c8385c\": container with ID starting with b284f7f15e12f58212aea896482f7f499bb735e67b80936c011ef81a20c8385c not found: ID does not exist" Jul 29 09:27:08.353779 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.353753 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx"] Jul 29 09:27:08.360882 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:08.360858 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-306d65/test-model-306d65-kserve-7dffffcd85-p2tdx"] Jul 29 09:27:09.362858 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:09.362829 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d904ef5-2210-48f3-8792-002854ed66b1" path="/var/lib/kubelet/pods/0d904ef5-2210-48f3-8792-002854ed66b1/volumes" Jul 29 09:27:09.415186 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:09.415156 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:27:24.514786 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:24.514700 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:27:38.532601 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:27:38.532568 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:28:07.407704 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:28:07.407668 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:28:46.012452 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:28:46.012349 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:29:11.213249 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:29:11.213210 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:29:22.711556 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:29:22.711522 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:29:52.115273 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:29:52.115240 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:30:36.119836 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:30:36.119797 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:31:06.515998 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:31:06.515960 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:31:31.153238 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:31:31.153155 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:31:57.223907 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:31:57.223867 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:32:14.317678 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:32:14.317640 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:32:36.817655 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:32:36.817617 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:32:55.313102 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:32:55.313058 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:33:17.224207 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:33:17.224114 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:33:36.615709 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:33:36.615667 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:33:58.219776 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:33:58.219735 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:34:17.363167 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:34:17.363128 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:34:40.037064 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:34:40.036963 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:34:57.828175 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:34:57.828143 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:35:20.662023 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:20.661991 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:35:34.773215 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.773180 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv"] Jul 29 09:35:34.773629 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.773542 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d904ef5-2210-48f3-8792-002854ed66b1" containerName="main" Jul 29 09:35:34.773629 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.773553 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d904ef5-2210-48f3-8792-002854ed66b1" containerName="main" Jul 29 09:35:34.773629 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.773626 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d904ef5-2210-48f3-8792-002854ed66b1" containerName="main" Jul 29 09:35:34.776637 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.776622 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:34.780737 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.780707 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0abb48\"/\"default-dockercfg-4b4wf\"" Jul 29 09:35:34.780737 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.780726 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0abb48\"/\"openshift-service-ca.crt\"" Jul 29 09:35:34.780907 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.780751 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0abb48\"/\"kube-root-ca.crt\"" Jul 29 09:35:34.780907 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.780800 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0abb48\"/\"sub-test-model-0abb48-kserve-self-signed-certs\"" Jul 29 09:35:34.839873 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.839839 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv"] Jul 29 09:35:34.917426 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.917393 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-model-cache\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:34.917426 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.917429 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-home\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:34.917687 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.917451 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-tmp-dir\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:34.917687 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.917542 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-tls-certs\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:34.917687 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.917598 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfp87\" (UniqueName: \"kubernetes.io/projected/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-kube-api-access-qfp87\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:34.917687 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:34.917660 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-dshm\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:35.018735 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.018637 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-tls-certs\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:35.018735 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.018687 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfp87\" (UniqueName: \"kubernetes.io/projected/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-kube-api-access-qfp87\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:35.018735 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.018729 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-dshm\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:35.019023 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.018784 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-model-cache\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:35.019023 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.018808 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-home\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:35.019023 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.018831 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-tmp-dir\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:35.019264 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.019238 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-model-cache\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:35.019387 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.019253 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-tmp-dir\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:35.019450 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.019386 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-home\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:35.021043 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.021020 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-dshm\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:35.021409 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.021393 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-tls-certs\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:35.041968 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.041932 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfp87\" (UniqueName: \"kubernetes.io/projected/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-kube-api-access-qfp87\") pod \"sub-test-model-0abb48-kserve-6696fddd9-sdsrv\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:35.085518 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.085484 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:35.233520 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.233494 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv"] Jul 29 09:35:35.235241 ip-10-0-141-66 kubenswrapper[2563]: W0729 09:35:35.235211 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfc2596c_0ce5_4923_ad36_c8a78a9eda07.slice/crio-bb20a3e475546a644573fdc8da3fd896c0b840d3480b1cad014d5e12f1b022a3 WatchSource:0}: Error finding container bb20a3e475546a644573fdc8da3fd896c0b840d3480b1cad014d5e12f1b022a3: Status 404 returned error can't find the container with id bb20a3e475546a644573fdc8da3fd896c0b840d3480b1cad014d5e12f1b022a3 Jul 29 09:35:35.744828 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.744795 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" event={"ID":"cfc2596c-0ce5-4923-ad36-c8a78a9eda07","Type":"ContainerStarted","Data":"bed152d8a4202096ef78d6b3410617b57c8fccb90a533c6b0769d230abec0b63"} Jul 29 09:35:35.744828 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.744834 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" event={"ID":"cfc2596c-0ce5-4923-ad36-c8a78a9eda07","Type":"ContainerStarted","Data":"bb20a3e475546a644573fdc8da3fd896c0b840d3480b1cad014d5e12f1b022a3"} Jul 29 09:35:35.774842 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:35.774790 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" podStartSLOduration=1.774769169 podStartE2EDuration="1.774769169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 09:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 09:35:35.772836553 +0000 UTC m=+2848.896937352" watchObservedRunningTime="2026-07-29 09:35:35.774769169 +0000 UTC m=+2848.898869969" Jul 29 09:35:38.422380 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:38.422332 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:35:45.086561 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:45.086517 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:45.086967 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:45.086583 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:45.099134 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:45.099105 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:35:45.803373 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:35:45.803339 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:36:13.016790 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:36:13.016710 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:37:00.121884 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:00.121849 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:37:17.715132 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:17.715091 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:37:30.396593 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:30.396507 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv"] Jul 29 09:37:33.315007 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:33.314970 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:37:45.415726 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.415666 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" podUID="cfc2596c-0ce5-4923-ad36-c8a78a9eda07" containerName="main" containerID="cri-o://bed152d8a4202096ef78d6b3410617b57c8fccb90a533c6b0769d230abec0b63" gracePeriod=45 Jul 29 09:37:45.661662 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.661637 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:37:45.730434 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.730329 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-dshm\") pod \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " Jul 29 09:37:45.730434 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.730432 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-tls-certs\") pod \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " Jul 29 09:37:45.730658 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.730463 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfp87\" (UniqueName: \"kubernetes.io/projected/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-kube-api-access-qfp87\") pod \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " Jul 29 09:37:45.730658 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.730485 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-home\") pod \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " Jul 29 09:37:45.730658 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.730508 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-model-cache\") pod \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " Jul 29 09:37:45.730658 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.730526 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-tmp-dir\") pod \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\" (UID: \"cfc2596c-0ce5-4923-ad36-c8a78a9eda07\") " Jul 29 09:37:45.730877 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.730795 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfc2596c-0ce5-4923-ad36-c8a78a9eda07" (UID: "cfc2596c-0ce5-4923-ad36-c8a78a9eda07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:37:45.730877 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.730802 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-home" (OuterVolumeSpecName: "home") pod "cfc2596c-0ce5-4923-ad36-c8a78a9eda07" (UID: "cfc2596c-0ce5-4923-ad36-c8a78a9eda07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:37:45.730877 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.730848 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfc2596c-0ce5-4923-ad36-c8a78a9eda07" (UID: "cfc2596c-0ce5-4923-ad36-c8a78a9eda07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:37:45.732582 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.732557 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-dshm" (OuterVolumeSpecName: "dshm") pod "cfc2596c-0ce5-4923-ad36-c8a78a9eda07" (UID: "cfc2596c-0ce5-4923-ad36-c8a78a9eda07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:37:45.732714 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.732631 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfc2596c-0ce5-4923-ad36-c8a78a9eda07" (UID: "cfc2596c-0ce5-4923-ad36-c8a78a9eda07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 09:37:45.732714 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.732668 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-kube-api-access-qfp87" (OuterVolumeSpecName: "kube-api-access-qfp87") pod "cfc2596c-0ce5-4923-ad36-c8a78a9eda07" (UID: "cfc2596c-0ce5-4923-ad36-c8a78a9eda07"). InnerVolumeSpecName "kube-api-access-qfp87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 09:37:45.831510 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.831483 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:37:45.831510 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.831507 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfp87\" (UniqueName: \"kubernetes.io/projected/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-kube-api-access-qfp87\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:37:45.831510 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.831517 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:37:45.831763 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.831525 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:37:45.831763 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.831534 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:37:45.831763 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:45.831541 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc2596c-0ce5-4923-ad36-c8a78a9eda07-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:37:46.361914 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:46.361880 2563 generic.go:356] "Generic (PLEG): container finished" podID="cfc2596c-0ce5-4923-ad36-c8a78a9eda07" containerID="bed152d8a4202096ef78d6b3410617b57c8fccb90a533c6b0769d230abec0b63" exitCode=0 Jul 29 09:37:46.362089 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:46.361926 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" Jul 29 09:37:46.362089 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:46.361961 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" event={"ID":"cfc2596c-0ce5-4923-ad36-c8a78a9eda07","Type":"ContainerDied","Data":"bed152d8a4202096ef78d6b3410617b57c8fccb90a533c6b0769d230abec0b63"} Jul 29 09:37:46.362089 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:46.362003 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv" event={"ID":"cfc2596c-0ce5-4923-ad36-c8a78a9eda07","Type":"ContainerDied","Data":"bb20a3e475546a644573fdc8da3fd896c0b840d3480b1cad014d5e12f1b022a3"} Jul 29 09:37:46.362089 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:46.362025 2563 scope.go:117] "RemoveContainer" containerID="bed152d8a4202096ef78d6b3410617b57c8fccb90a533c6b0769d230abec0b63" Jul 29 09:37:46.371170 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:46.371152 2563 scope.go:117] "RemoveContainer" containerID="bed152d8a4202096ef78d6b3410617b57c8fccb90a533c6b0769d230abec0b63" Jul 29 09:37:46.371432 ip-10-0-141-66 kubenswrapper[2563]: E0729 09:37:46.371408 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed152d8a4202096ef78d6b3410617b57c8fccb90a533c6b0769d230abec0b63\": container with ID starting with bed152d8a4202096ef78d6b3410617b57c8fccb90a533c6b0769d230abec0b63 not found: ID does not exist" containerID="bed152d8a4202096ef78d6b3410617b57c8fccb90a533c6b0769d230abec0b63" Jul 29 09:37:46.371489 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:46.371442 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed152d8a4202096ef78d6b3410617b57c8fccb90a533c6b0769d230abec0b63"} err="failed to get container status \"bed152d8a4202096ef78d6b3410617b57c8fccb90a533c6b0769d230abec0b63\": rpc error: code = NotFound desc = could not find container \"bed152d8a4202096ef78d6b3410617b57c8fccb90a533c6b0769d230abec0b63\": container with ID starting with bed152d8a4202096ef78d6b3410617b57c8fccb90a533c6b0769d230abec0b63 not found: ID does not exist" Jul 29 09:37:46.385828 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:46.385799 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv"] Jul 29 09:37:46.391613 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:46.391593 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0abb48/sub-test-model-0abb48-kserve-6696fddd9-sdsrv"] Jul 29 09:37:47.362139 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:47.362106 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfc2596c-0ce5-4923-ad36-c8a78a9eda07" path="/var/lib/kubelet/pods/cfc2596c-0ce5-4923-ad36-c8a78a9eda07/volumes" Jul 29 09:37:51.919844 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:37:51.919809 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:38:03.721216 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:38:03.721181 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:38:39.418402 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:38:39.418356 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:39:00.316082 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:00.316000 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:39:08.592594 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:08.592562 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7783-139681293146672"] Jul 29 09:39:08.593049 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:08.592930 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc2596c-0ce5-4923-ad36-c8a78a9eda07" containerName="main" Jul 29 09:39:08.593049 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:08.592941 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc2596c-0ce5-4923-ad36-c8a78a9eda07" containerName="main" Jul 29 09:39:08.593049 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:08.593004 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfc2596c-0ce5-4923-ad36-c8a78a9eda07" containerName="main" Jul 29 09:39:08.595977 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:08.595962 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293146672" Jul 29 09:39:08.598640 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:08.598620 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rmth9\"" Jul 29 09:39:08.603983 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:08.603963 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7783-139681293146672"] Jul 29 09:39:08.740167 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:08.740135 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tv8t\" (UniqueName: \"kubernetes.io/projected/28bf5564-6fee-4898-87ce-093f32178c7f-kube-api-access-4tv8t\") pod \"test-curl-7783-139681293146672\" (UID: \"28bf5564-6fee-4898-87ce-093f32178c7f\") " pod="opendatahub/test-curl-7783-139681293146672" Jul 29 09:39:08.840767 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:08.840734 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tv8t\" (UniqueName: \"kubernetes.io/projected/28bf5564-6fee-4898-87ce-093f32178c7f-kube-api-access-4tv8t\") pod \"test-curl-7783-139681293146672\" (UID: \"28bf5564-6fee-4898-87ce-093f32178c7f\") " pod="opendatahub/test-curl-7783-139681293146672" Jul 29 09:39:08.848921 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:08.848843 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tv8t\" (UniqueName: \"kubernetes.io/projected/28bf5564-6fee-4898-87ce-093f32178c7f-kube-api-access-4tv8t\") pod \"test-curl-7783-139681293146672\" (UID: \"28bf5564-6fee-4898-87ce-093f32178c7f\") " pod="opendatahub/test-curl-7783-139681293146672" Jul 29 09:39:08.906005 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:08.905977 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293146672" Jul 29 09:39:09.039467 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:09.039437 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7783-139681293146672"] Jul 29 09:39:09.041817 ip-10-0-141-66 kubenswrapper[2563]: W0729 09:39:09.041789 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28bf5564_6fee_4898_87ce_093f32178c7f.slice/crio-4d9135a99156afc1fc488bef17331ec336c6af1d2a1fffaa3e4acdcedd8e043a WatchSource:0}: Error finding container 4d9135a99156afc1fc488bef17331ec336c6af1d2a1fffaa3e4acdcedd8e043a: Status 404 returned error can't find the container with id 4d9135a99156afc1fc488bef17331ec336c6af1d2a1fffaa3e4acdcedd8e043a Jul 29 09:39:09.043186 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:09.043166 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 09:39:09.753246 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:09.753217 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7783-139681293146672" event={"ID":"28bf5564-6fee-4898-87ce-093f32178c7f","Type":"ContainerStarted","Data":"4d9135a99156afc1fc488bef17331ec336c6af1d2a1fffaa3e4acdcedd8e043a"} Jul 29 09:39:10.761523 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:10.761486 2563 generic.go:356] "Generic (PLEG): container finished" podID="28bf5564-6fee-4898-87ce-093f32178c7f" containerID="c55a06aa587eb3a57673261586ddcd1272942de74c3368c175d4b8c856e110d5" exitCode=0 Jul 29 09:39:10.761523 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:10.761521 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7783-139681293146672" event={"ID":"28bf5564-6fee-4898-87ce-093f32178c7f","Type":"ContainerDied","Data":"c55a06aa587eb3a57673261586ddcd1272942de74c3368c175d4b8c856e110d5"} Jul 29 09:39:11.897133 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:11.897104 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293146672" Jul 29 09:39:11.933530 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:11.933497 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7783-139681293146672_28bf5564-6fee-4898-87ce-093f32178c7f/test-curl-7783-139681293146672/0.log" Jul 29 09:39:11.953255 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:11.953224 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7783-139681293146672"] Jul 29 09:39:11.958700 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:11.958678 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7783-139681293146672"] Jul 29 09:39:11.967201 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:11.967183 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tv8t\" (UniqueName: \"kubernetes.io/projected/28bf5564-6fee-4898-87ce-093f32178c7f-kube-api-access-4tv8t\") pod \"28bf5564-6fee-4898-87ce-093f32178c7f\" (UID: \"28bf5564-6fee-4898-87ce-093f32178c7f\") " Jul 29 09:39:11.969301 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:11.969269 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28bf5564-6fee-4898-87ce-093f32178c7f-kube-api-access-4tv8t" (OuterVolumeSpecName: "kube-api-access-4tv8t") pod "28bf5564-6fee-4898-87ce-093f32178c7f" (UID: "28bf5564-6fee-4898-87ce-093f32178c7f"). InnerVolumeSpecName "kube-api-access-4tv8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 09:39:12.068648 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:12.068561 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tv8t\" (UniqueName: \"kubernetes.io/projected/28bf5564-6fee-4898-87ce-093f32178c7f-kube-api-access-4tv8t\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:39:12.109507 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:12.109473 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7783-139681293136432"] Jul 29 09:39:12.109827 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:12.109815 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28bf5564-6fee-4898-87ce-093f32178c7f" containerName="test-curl-7783-139681293146672" Jul 29 09:39:12.109873 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:12.109829 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="28bf5564-6fee-4898-87ce-093f32178c7f" containerName="test-curl-7783-139681293146672" Jul 29 09:39:12.109916 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:12.109895 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="28bf5564-6fee-4898-87ce-093f32178c7f" containerName="test-curl-7783-139681293146672" Jul 29 09:39:12.112797 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:12.112780 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293136432" Jul 29 09:39:12.118187 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:12.118165 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7783-139681293136432"] Jul 29 09:39:12.271116 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:12.271075 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh4lf\" (UniqueName: \"kubernetes.io/projected/9595cd6d-2a62-41c1-b770-2b66a8a8da27-kube-api-access-qh4lf\") pod \"test-curl-7783-139681293136432\" (UID: \"9595cd6d-2a62-41c1-b770-2b66a8a8da27\") " pod="opendatahub/test-curl-7783-139681293136432" Jul 29 09:39:12.371959 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:12.371860 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh4lf\" (UniqueName: \"kubernetes.io/projected/9595cd6d-2a62-41c1-b770-2b66a8a8da27-kube-api-access-qh4lf\") pod \"test-curl-7783-139681293136432\" (UID: \"9595cd6d-2a62-41c1-b770-2b66a8a8da27\") " pod="opendatahub/test-curl-7783-139681293136432" Jul 29 09:39:12.380682 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:12.380655 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh4lf\" (UniqueName: \"kubernetes.io/projected/9595cd6d-2a62-41c1-b770-2b66a8a8da27-kube-api-access-qh4lf\") pod \"test-curl-7783-139681293136432\" (UID: \"9595cd6d-2a62-41c1-b770-2b66a8a8da27\") " pod="opendatahub/test-curl-7783-139681293136432" Jul 29 09:39:12.423647 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:12.423618 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293136432" Jul 29 09:39:12.549558 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:12.549532 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7783-139681293136432"] Jul 29 09:39:12.551462 ip-10-0-141-66 kubenswrapper[2563]: W0729 09:39:12.551433 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9595cd6d_2a62_41c1_b770_2b66a8a8da27.slice/crio-8b9212b1b36f135b402785b22747e29714c5dfba283b59b596cc76e0e13f80bb WatchSource:0}: Error finding container 8b9212b1b36f135b402785b22747e29714c5dfba283b59b596cc76e0e13f80bb: Status 404 returned error can't find the container with id 8b9212b1b36f135b402785b22747e29714c5dfba283b59b596cc76e0e13f80bb Jul 29 09:39:12.772818 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:12.772787 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d9135a99156afc1fc488bef17331ec336c6af1d2a1fffaa3e4acdcedd8e043a" Jul 29 09:39:12.772818 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:12.772792 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293146672" Jul 29 09:39:12.774000 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:12.773973 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7783-139681293136432" event={"ID":"9595cd6d-2a62-41c1-b770-2b66a8a8da27","Type":"ContainerStarted","Data":"8b9212b1b36f135b402785b22747e29714c5dfba283b59b596cc76e0e13f80bb"} Jul 29 09:39:12.783483 ip-10-0-141-66 kubenswrapper[2563]: E0729 09:39:12.783458 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7783-139681293146672\" is forbidden: User \"system:node:ip-10-0-141-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-66.ec2.internal' and this object" podUID="28bf5564-6fee-4898-87ce-093f32178c7f" pod="opendatahub/test-curl-7783-139681293146672" Jul 29 09:39:13.362955 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:13.362923 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28bf5564-6fee-4898-87ce-093f32178c7f" path="/var/lib/kubelet/pods/28bf5564-6fee-4898-87ce-093f32178c7f/volumes" Jul 29 09:39:13.780507 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:13.780469 2563 generic.go:356] "Generic (PLEG): container finished" podID="9595cd6d-2a62-41c1-b770-2b66a8a8da27" containerID="45d33f9357f3b169a457eced2466c7b26dd11972723f1b32aca63af166f32a8e" exitCode=0 Jul 29 09:39:13.780672 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:13.780542 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7783-139681293136432" event={"ID":"9595cd6d-2a62-41c1-b770-2b66a8a8da27","Type":"ContainerDied","Data":"45d33f9357f3b169a457eced2466c7b26dd11972723f1b32aca63af166f32a8e"} Jul 29 09:39:14.916690 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:14.916666 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293136432" Jul 29 09:39:14.936021 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:14.935996 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7783-139681293136432_9595cd6d-2a62-41c1-b770-2b66a8a8da27/test-curl-7783-139681293136432/0.log" Jul 29 09:39:14.956702 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:14.956677 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7783-139681293136432"] Jul 29 09:39:14.960015 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:14.959992 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7783-139681293136432"] Jul 29 09:39:14.994533 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:14.994511 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh4lf\" (UniqueName: \"kubernetes.io/projected/9595cd6d-2a62-41c1-b770-2b66a8a8da27-kube-api-access-qh4lf\") pod \"9595cd6d-2a62-41c1-b770-2b66a8a8da27\" (UID: \"9595cd6d-2a62-41c1-b770-2b66a8a8da27\") " Jul 29 09:39:14.996521 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:14.996498 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9595cd6d-2a62-41c1-b770-2b66a8a8da27-kube-api-access-qh4lf" (OuterVolumeSpecName: "kube-api-access-qh4lf") pod "9595cd6d-2a62-41c1-b770-2b66a8a8da27" (UID: "9595cd6d-2a62-41c1-b770-2b66a8a8da27"). InnerVolumeSpecName "kube-api-access-qh4lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 09:39:15.096146 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.096062 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh4lf\" (UniqueName: \"kubernetes.io/projected/9595cd6d-2a62-41c1-b770-2b66a8a8da27-kube-api-access-qh4lf\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:39:15.113654 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.113627 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7783-139681293117456"] Jul 29 09:39:15.113973 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.113960 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9595cd6d-2a62-41c1-b770-2b66a8a8da27" containerName="test-curl-7783-139681293136432" Jul 29 09:39:15.114029 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.113974 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9595cd6d-2a62-41c1-b770-2b66a8a8da27" containerName="test-curl-7783-139681293136432" Jul 29 09:39:15.114072 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.114047 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9595cd6d-2a62-41c1-b770-2b66a8a8da27" containerName="test-curl-7783-139681293136432" Jul 29 09:39:15.116862 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.116847 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293117456" Jul 29 09:39:15.124214 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.123977 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7783-139681293117456"] Jul 29 09:39:15.197170 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.197133 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlvgg\" (UniqueName: \"kubernetes.io/projected/3a2af7fe-1f12-44e9-ba02-ca2c622b94e4-kube-api-access-xlvgg\") pod \"test-curl-7783-139681293117456\" (UID: \"3a2af7fe-1f12-44e9-ba02-ca2c622b94e4\") " pod="opendatahub/test-curl-7783-139681293117456" Jul 29 09:39:15.298114 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.298074 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlvgg\" (UniqueName: \"kubernetes.io/projected/3a2af7fe-1f12-44e9-ba02-ca2c622b94e4-kube-api-access-xlvgg\") pod \"test-curl-7783-139681293117456\" (UID: \"3a2af7fe-1f12-44e9-ba02-ca2c622b94e4\") " pod="opendatahub/test-curl-7783-139681293117456" Jul 29 09:39:15.306753 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.306727 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlvgg\" (UniqueName: \"kubernetes.io/projected/3a2af7fe-1f12-44e9-ba02-ca2c622b94e4-kube-api-access-xlvgg\") pod \"test-curl-7783-139681293117456\" (UID: \"3a2af7fe-1f12-44e9-ba02-ca2c622b94e4\") " pod="opendatahub/test-curl-7783-139681293117456" Jul 29 09:39:15.361954 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.361881 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9595cd6d-2a62-41c1-b770-2b66a8a8da27" path="/var/lib/kubelet/pods/9595cd6d-2a62-41c1-b770-2b66a8a8da27/volumes" Jul 29 09:39:15.427885 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.427835 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293117456" Jul 29 09:39:15.556546 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.556518 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7783-139681293117456"] Jul 29 09:39:15.558454 ip-10-0-141-66 kubenswrapper[2563]: W0729 09:39:15.558426 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a2af7fe_1f12_44e9_ba02_ca2c622b94e4.slice/crio-c87fcfc9ec0062d3db5e96c2d5a9f792978cd69e7382f120d24f4fe9e488a833 WatchSource:0}: Error finding container c87fcfc9ec0062d3db5e96c2d5a9f792978cd69e7382f120d24f4fe9e488a833: Status 404 returned error can't find the container with id c87fcfc9ec0062d3db5e96c2d5a9f792978cd69e7382f120d24f4fe9e488a833 Jul 29 09:39:15.791111 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.791076 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7783-139681293117456" event={"ID":"3a2af7fe-1f12-44e9-ba02-ca2c622b94e4","Type":"ContainerStarted","Data":"c87fcfc9ec0062d3db5e96c2d5a9f792978cd69e7382f120d24f4fe9e488a833"} Jul 29 09:39:15.792266 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.792235 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293136432" Jul 29 09:39:15.792433 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:15.792246 2563 scope.go:117] "RemoveContainer" containerID="45d33f9357f3b169a457eced2466c7b26dd11972723f1b32aca63af166f32a8e" Jul 29 09:39:16.799346 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:16.799313 2563 generic.go:356] "Generic (PLEG): container finished" podID="3a2af7fe-1f12-44e9-ba02-ca2c622b94e4" containerID="52c746186dc8a402ba0ead8b745cdc2e61a9218ada046775726d58e66aea0e1c" exitCode=0 Jul 29 09:39:16.799815 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:16.799406 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7783-139681293117456" event={"ID":"3a2af7fe-1f12-44e9-ba02-ca2c622b94e4","Type":"ContainerDied","Data":"52c746186dc8a402ba0ead8b745cdc2e61a9218ada046775726d58e66aea0e1c"} Jul 29 09:39:17.941095 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:17.941067 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293117456" Jul 29 09:39:17.960200 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:17.960172 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7783-139681293117456_3a2af7fe-1f12-44e9-ba02-ca2c622b94e4/test-curl-7783-139681293117456/0.log" Jul 29 09:39:17.978506 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:17.978482 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7783-139681293117456"] Jul 29 09:39:17.984136 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:17.984112 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7783-139681293117456"] Jul 29 09:39:18.013956 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.013926 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:39:18.023570 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.023545 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlvgg\" (UniqueName: \"kubernetes.io/projected/3a2af7fe-1f12-44e9-ba02-ca2c622b94e4-kube-api-access-xlvgg\") pod \"3a2af7fe-1f12-44e9-ba02-ca2c622b94e4\" (UID: \"3a2af7fe-1f12-44e9-ba02-ca2c622b94e4\") " Jul 29 09:39:18.025633 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.025612 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a2af7fe-1f12-44e9-ba02-ca2c622b94e4-kube-api-access-xlvgg" (OuterVolumeSpecName: "kube-api-access-xlvgg") pod "3a2af7fe-1f12-44e9-ba02-ca2c622b94e4" (UID: "3a2af7fe-1f12-44e9-ba02-ca2c622b94e4"). InnerVolumeSpecName "kube-api-access-xlvgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 09:39:18.106694 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.106618 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7783-139681293117456"] Jul 29 09:39:18.106977 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.106963 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a2af7fe-1f12-44e9-ba02-ca2c622b94e4" containerName="test-curl-7783-139681293117456" Jul 29 09:39:18.107046 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.106979 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a2af7fe-1f12-44e9-ba02-ca2c622b94e4" containerName="test-curl-7783-139681293117456" Jul 29 09:39:18.107046 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.107041 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a2af7fe-1f12-44e9-ba02-ca2c622b94e4" containerName="test-curl-7783-139681293117456" Jul 29 09:39:18.109865 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.109848 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293117456" Jul 29 09:39:18.115259 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.115224 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7783-139681293117456"] Jul 29 09:39:18.124350 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.124324 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlvgg\" (UniqueName: \"kubernetes.io/projected/3a2af7fe-1f12-44e9-ba02-ca2c622b94e4-kube-api-access-xlvgg\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:39:18.225264 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.225234 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x88n\" (UniqueName: \"kubernetes.io/projected/9dd36888-6437-4457-8e62-90344a6b5571-kube-api-access-2x88n\") pod \"test-curl-7783-139681293117456\" (UID: \"9dd36888-6437-4457-8e62-90344a6b5571\") " pod="opendatahub/test-curl-7783-139681293117456" Jul 29 09:39:18.326643 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.326607 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x88n\" (UniqueName: \"kubernetes.io/projected/9dd36888-6437-4457-8e62-90344a6b5571-kube-api-access-2x88n\") pod \"test-curl-7783-139681293117456\" (UID: \"9dd36888-6437-4457-8e62-90344a6b5571\") " pod="opendatahub/test-curl-7783-139681293117456" Jul 29 09:39:18.335739 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.335712 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x88n\" (UniqueName: \"kubernetes.io/projected/9dd36888-6437-4457-8e62-90344a6b5571-kube-api-access-2x88n\") pod \"test-curl-7783-139681293117456\" (UID: \"9dd36888-6437-4457-8e62-90344a6b5571\") " pod="opendatahub/test-curl-7783-139681293117456" Jul 29 09:39:18.419873 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.419843 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293117456" Jul 29 09:39:18.551322 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.547319 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7783-139681293117456"] Jul 29 09:39:18.554240 ip-10-0-141-66 kubenswrapper[2563]: W0729 09:39:18.554191 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dd36888_6437_4457_8e62_90344a6b5571.slice/crio-05df41da0af2dbe435155fd1c128a566fe26511205d042f8c5b28d6e4cf0bd5c WatchSource:0}: Error finding container 05df41da0af2dbe435155fd1c128a566fe26511205d042f8c5b28d6e4cf0bd5c: Status 404 returned error can't find the container with id 05df41da0af2dbe435155fd1c128a566fe26511205d042f8c5b28d6e4cf0bd5c Jul 29 09:39:18.810817 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.810736 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7783-139681293117456" event={"ID":"9dd36888-6437-4457-8e62-90344a6b5571","Type":"ContainerStarted","Data":"05df41da0af2dbe435155fd1c128a566fe26511205d042f8c5b28d6e4cf0bd5c"} Jul 29 09:39:18.811994 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.811969 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293117456" Jul 29 09:39:18.812118 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.812003 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c87fcfc9ec0062d3db5e96c2d5a9f792978cd69e7382f120d24f4fe9e488a833" Jul 29 09:39:18.823969 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:18.823940 2563 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7783-139681293117456" oldPodUID="3a2af7fe-1f12-44e9-ba02-ca2c622b94e4" podUID="9dd36888-6437-4457-8e62-90344a6b5571" Jul 29 09:39:19.363072 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:19.363040 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a2af7fe-1f12-44e9-ba02-ca2c622b94e4" path="/var/lib/kubelet/pods/3a2af7fe-1f12-44e9-ba02-ca2c622b94e4/volumes" Jul 29 09:39:19.818373 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:19.818329 2563 generic.go:356] "Generic (PLEG): container finished" podID="9dd36888-6437-4457-8e62-90344a6b5571" containerID="f4126d3210e6557faf450d2242fcdc83bda41f8c79bd86f9b7b89e8d42ff217d" exitCode=0 Jul 29 09:39:19.818567 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:19.818415 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7783-139681293117456" event={"ID":"9dd36888-6437-4457-8e62-90344a6b5571","Type":"ContainerDied","Data":"f4126d3210e6557faf450d2242fcdc83bda41f8c79bd86f9b7b89e8d42ff217d"} Jul 29 09:39:20.959576 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:20.959553 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293117456" Jul 29 09:39:20.977209 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:20.977183 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7783-139681293117456_9dd36888-6437-4457-8e62-90344a6b5571/test-curl-7783-139681293117456/0.log" Jul 29 09:39:20.998412 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:20.998385 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7783-139681293117456"] Jul 29 09:39:21.003297 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.003267 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7783-139681293117456"] Jul 29 09:39:21.051712 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.051684 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2x88n\" (UniqueName: \"kubernetes.io/projected/9dd36888-6437-4457-8e62-90344a6b5571-kube-api-access-2x88n\") pod \"9dd36888-6437-4457-8e62-90344a6b5571\" (UID: \"9dd36888-6437-4457-8e62-90344a6b5571\") " Jul 29 09:39:21.053725 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.053697 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd36888-6437-4457-8e62-90344a6b5571-kube-api-access-2x88n" (OuterVolumeSpecName: "kube-api-access-2x88n") pod "9dd36888-6437-4457-8e62-90344a6b5571" (UID: "9dd36888-6437-4457-8e62-90344a6b5571"). InnerVolumeSpecName "kube-api-access-2x88n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 09:39:21.131303 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.131230 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7783-139681293118464"] Jul 29 09:39:21.131601 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.131588 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd36888-6437-4457-8e62-90344a6b5571" containerName="test-curl-7783-139681293117456" Jul 29 09:39:21.131646 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.131602 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd36888-6437-4457-8e62-90344a6b5571" containerName="test-curl-7783-139681293117456" Jul 29 09:39:21.131691 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.131681 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dd36888-6437-4457-8e62-90344a6b5571" containerName="test-curl-7783-139681293117456" Jul 29 09:39:21.134520 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.134505 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293118464" Jul 29 09:39:21.141213 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.141188 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7783-139681293118464"] Jul 29 09:39:21.152981 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.152958 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x88n\" (UniqueName: \"kubernetes.io/projected/9dd36888-6437-4457-8e62-90344a6b5571-kube-api-access-2x88n\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:39:21.254217 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.254182 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qkck\" (UniqueName: \"kubernetes.io/projected/4efa3da5-53e6-488b-b7ae-7efed90204cb-kube-api-access-2qkck\") pod \"test-curl-7783-139681293118464\" (UID: \"4efa3da5-53e6-488b-b7ae-7efed90204cb\") " pod="opendatahub/test-curl-7783-139681293118464" Jul 29 09:39:21.355036 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.354999 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qkck\" (UniqueName: \"kubernetes.io/projected/4efa3da5-53e6-488b-b7ae-7efed90204cb-kube-api-access-2qkck\") pod \"test-curl-7783-139681293118464\" (UID: \"4efa3da5-53e6-488b-b7ae-7efed90204cb\") " pod="opendatahub/test-curl-7783-139681293118464" Jul 29 09:39:21.362205 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.362173 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dd36888-6437-4457-8e62-90344a6b5571" path="/var/lib/kubelet/pods/9dd36888-6437-4457-8e62-90344a6b5571/volumes" Jul 29 09:39:21.364104 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.364075 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qkck\" (UniqueName: \"kubernetes.io/projected/4efa3da5-53e6-488b-b7ae-7efed90204cb-kube-api-access-2qkck\") pod \"test-curl-7783-139681293118464\" (UID: \"4efa3da5-53e6-488b-b7ae-7efed90204cb\") " pod="opendatahub/test-curl-7783-139681293118464" Jul 29 09:39:21.445104 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.445071 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293118464" Jul 29 09:39:21.793624 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.793599 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7783-139681293118464"] Jul 29 09:39:21.795268 ip-10-0-141-66 kubenswrapper[2563]: W0729 09:39:21.795229 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4efa3da5_53e6_488b_b7ae_7efed90204cb.slice/crio-6565be63a160081d0dc5b1bcca2fc6f1c209b7e3e9f7f12f0f3550a02bafbaf1 WatchSource:0}: Error finding container 6565be63a160081d0dc5b1bcca2fc6f1c209b7e3e9f7f12f0f3550a02bafbaf1: Status 404 returned error can't find the container with id 6565be63a160081d0dc5b1bcca2fc6f1c209b7e3e9f7f12f0f3550a02bafbaf1 Jul 29 09:39:21.830928 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.830895 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7783-139681293118464" event={"ID":"4efa3da5-53e6-488b-b7ae-7efed90204cb","Type":"ContainerStarted","Data":"6565be63a160081d0dc5b1bcca2fc6f1c209b7e3e9f7f12f0f3550a02bafbaf1"} Jul 29 09:39:21.832197 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.832178 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293117456" Jul 29 09:39:21.832289 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:21.832193 2563 scope.go:117] "RemoveContainer" containerID="f4126d3210e6557faf450d2242fcdc83bda41f8c79bd86f9b7b89e8d42ff217d" Jul 29 09:39:22.839216 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:22.839176 2563 generic.go:356] "Generic (PLEG): container finished" podID="4efa3da5-53e6-488b-b7ae-7efed90204cb" containerID="4fc8622b9c650a0adbfd2b5fb8a939f764350d989a25618a9a97e1a95c6a5187" exitCode=0 Jul 29 09:39:22.839621 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:22.839221 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7783-139681293118464" event={"ID":"4efa3da5-53e6-488b-b7ae-7efed90204cb","Type":"ContainerDied","Data":"4fc8622b9c650a0adbfd2b5fb8a939f764350d989a25618a9a97e1a95c6a5187"} Jul 29 09:39:23.983803 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:23.983776 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293118464" Jul 29 09:39:24.003809 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:24.003778 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7783-139681293118464_4efa3da5-53e6-488b-b7ae-7efed90204cb/test-curl-7783-139681293118464/0.log" Jul 29 09:39:24.026083 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:24.026017 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7783-139681293118464"] Jul 29 09:39:24.028031 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:24.028008 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7783-139681293118464"] Jul 29 09:39:24.079412 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:24.079381 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qkck\" (UniqueName: \"kubernetes.io/projected/4efa3da5-53e6-488b-b7ae-7efed90204cb-kube-api-access-2qkck\") pod \"4efa3da5-53e6-488b-b7ae-7efed90204cb\" (UID: \"4efa3da5-53e6-488b-b7ae-7efed90204cb\") " Jul 29 09:39:24.081385 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:24.081352 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efa3da5-53e6-488b-b7ae-7efed90204cb-kube-api-access-2qkck" (OuterVolumeSpecName: "kube-api-access-2qkck") pod "4efa3da5-53e6-488b-b7ae-7efed90204cb" (UID: "4efa3da5-53e6-488b-b7ae-7efed90204cb"). InnerVolumeSpecName "kube-api-access-2qkck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 09:39:24.180764 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:24.180730 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qkck\" (UniqueName: \"kubernetes.io/projected/4efa3da5-53e6-488b-b7ae-7efed90204cb-kube-api-access-2qkck\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:39:24.849811 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:24.849780 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7783-139681293118464" Jul 29 09:39:24.849988 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:24.849824 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6565be63a160081d0dc5b1bcca2fc6f1c209b7e3e9f7f12f0f3550a02bafbaf1" Jul 29 09:39:24.859934 ip-10-0-141-66 kubenswrapper[2563]: E0729 09:39:24.859907 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7783-139681293118464\" is forbidden: User \"system:node:ip-10-0-141-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-66.ec2.internal' and this object" podUID="4efa3da5-53e6-488b-b7ae-7efed90204cb" pod="opendatahub/test-curl-7783-139681293118464" Jul 29 09:39:25.361887 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:25.361852 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4efa3da5-53e6-488b-b7ae-7efed90204cb" path="/var/lib/kubelet/pods/4efa3da5-53e6-488b-b7ae-7efed90204cb/volumes" Jul 29 09:39:41.315099 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:41.315056 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:39:43.158484 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:43.158443 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gll82"] Jul 29 09:39:43.158993 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:43.158973 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4efa3da5-53e6-488b-b7ae-7efed90204cb" containerName="test-curl-7783-139681293118464" Jul 29 09:39:43.159075 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:43.158995 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4efa3da5-53e6-488b-b7ae-7efed90204cb" containerName="test-curl-7783-139681293118464" Jul 29 09:39:43.159131 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:43.159103 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4efa3da5-53e6-488b-b7ae-7efed90204cb" containerName="test-curl-7783-139681293118464" Jul 29 09:39:43.164175 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:43.164153 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gll82" Jul 29 09:39:43.168284 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:43.168262 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-d744b\"" Jul 29 09:39:43.169971 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:43.169948 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gll82"] Jul 29 09:39:43.235313 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:43.235283 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hg8g\" (UniqueName: \"kubernetes.io/projected/319aaefc-deac-41b0-ab8f-1dfded39f59f-kube-api-access-6hg8g\") pod \"kuadrant-operator-catalog-gll82\" (UID: \"319aaefc-deac-41b0-ab8f-1dfded39f59f\") " pod="kuadrant-system/kuadrant-operator-catalog-gll82" Jul 29 09:39:43.336265 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:43.336220 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hg8g\" (UniqueName: \"kubernetes.io/projected/319aaefc-deac-41b0-ab8f-1dfded39f59f-kube-api-access-6hg8g\") pod \"kuadrant-operator-catalog-gll82\" (UID: \"319aaefc-deac-41b0-ab8f-1dfded39f59f\") " pod="kuadrant-system/kuadrant-operator-catalog-gll82" Jul 29 09:39:43.345910 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:43.345879 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hg8g\" (UniqueName: \"kubernetes.io/projected/319aaefc-deac-41b0-ab8f-1dfded39f59f-kube-api-access-6hg8g\") pod \"kuadrant-operator-catalog-gll82\" (UID: \"319aaefc-deac-41b0-ab8f-1dfded39f59f\") " pod="kuadrant-system/kuadrant-operator-catalog-gll82" Jul 29 09:39:43.474642 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:43.474607 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gll82" Jul 29 09:39:43.611435 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:43.611412 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gll82"] Jul 29 09:39:43.614102 ip-10-0-141-66 kubenswrapper[2563]: W0729 09:39:43.614064 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319aaefc_deac_41b0_ab8f_1dfded39f59f.slice/crio-383265ae954f96705a7021fb45ae98f01912a3a80865977cbe7d921b4a7e04c2 WatchSource:0}: Error finding container 383265ae954f96705a7021fb45ae98f01912a3a80865977cbe7d921b4a7e04c2: Status 404 returned error can't find the container with id 383265ae954f96705a7021fb45ae98f01912a3a80865977cbe7d921b4a7e04c2 Jul 29 09:39:43.945818 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:43.945783 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gll82" event={"ID":"319aaefc-deac-41b0-ab8f-1dfded39f59f","Type":"ContainerStarted","Data":"383265ae954f96705a7021fb45ae98f01912a3a80865977cbe7d921b4a7e04c2"} Jul 29 09:39:45.957506 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:45.957466 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gll82" event={"ID":"319aaefc-deac-41b0-ab8f-1dfded39f59f","Type":"ContainerStarted","Data":"8a89c5f9f41c4688dde5f59524224228d08d1e87cbf406ea4fd3bbde237c0174"} Jul 29 09:39:45.974095 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:45.974044 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-gll82" podStartSLOduration=1.2976274509999999 podStartE2EDuration="2.974027741s" totalImagesPullingTime="1.67640029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 09:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 09:39:45.972848173 +0000 UTC m=+3099.096948972" watchObservedRunningTime="2026-07-29 09:39:45.974027741 +0000 UTC m=+3099.098128539" Jul 29 09:39:53.475660 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:53.475617 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-gll82" Jul 29 09:39:53.476087 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:53.475700 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-gll82" Jul 29 09:39:53.497220 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:53.497189 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-gll82" Jul 29 09:39:54.018891 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:54.018853 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-gll82" Jul 29 09:39:54.048991 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:54.048957 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gll82"] Jul 29 09:39:56.005960 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:56.005919 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-gll82" podUID="319aaefc-deac-41b0-ab8f-1dfded39f59f" containerName="registry-server" containerID="cri-o://8a89c5f9f41c4688dde5f59524224228d08d1e87cbf406ea4fd3bbde237c0174" gracePeriod=2 Jul 29 09:39:56.268561 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:56.268503 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gll82" Jul 29 09:39:56.352863 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:56.352831 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hg8g\" (UniqueName: \"kubernetes.io/projected/319aaefc-deac-41b0-ab8f-1dfded39f59f-kube-api-access-6hg8g\") pod \"319aaefc-deac-41b0-ab8f-1dfded39f59f\" (UID: \"319aaefc-deac-41b0-ab8f-1dfded39f59f\") " Jul 29 09:39:56.354885 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:56.354853 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319aaefc-deac-41b0-ab8f-1dfded39f59f-kube-api-access-6hg8g" (OuterVolumeSpecName: "kube-api-access-6hg8g") pod "319aaefc-deac-41b0-ab8f-1dfded39f59f" (UID: "319aaefc-deac-41b0-ab8f-1dfded39f59f"). InnerVolumeSpecName "kube-api-access-6hg8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 09:39:56.454319 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:56.454284 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hg8g\" (UniqueName: \"kubernetes.io/projected/319aaefc-deac-41b0-ab8f-1dfded39f59f-kube-api-access-6hg8g\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:39:57.012115 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:57.012078 2563 generic.go:356] "Generic (PLEG): container finished" podID="319aaefc-deac-41b0-ab8f-1dfded39f59f" containerID="8a89c5f9f41c4688dde5f59524224228d08d1e87cbf406ea4fd3bbde237c0174" exitCode=0 Jul 29 09:39:57.012661 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:57.012133 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gll82" event={"ID":"319aaefc-deac-41b0-ab8f-1dfded39f59f","Type":"ContainerDied","Data":"8a89c5f9f41c4688dde5f59524224228d08d1e87cbf406ea4fd3bbde237c0174"} Jul 29 09:39:57.012661 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:57.012176 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gll82" event={"ID":"319aaefc-deac-41b0-ab8f-1dfded39f59f","Type":"ContainerDied","Data":"383265ae954f96705a7021fb45ae98f01912a3a80865977cbe7d921b4a7e04c2"} Jul 29 09:39:57.012661 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:57.012135 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gll82" Jul 29 09:39:57.012661 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:57.012202 2563 scope.go:117] "RemoveContainer" containerID="8a89c5f9f41c4688dde5f59524224228d08d1e87cbf406ea4fd3bbde237c0174" Jul 29 09:39:57.022433 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:57.022419 2563 scope.go:117] "RemoveContainer" containerID="8a89c5f9f41c4688dde5f59524224228d08d1e87cbf406ea4fd3bbde237c0174" Jul 29 09:39:57.022709 ip-10-0-141-66 kubenswrapper[2563]: E0729 09:39:57.022690 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a89c5f9f41c4688dde5f59524224228d08d1e87cbf406ea4fd3bbde237c0174\": container with ID starting with 8a89c5f9f41c4688dde5f59524224228d08d1e87cbf406ea4fd3bbde237c0174 not found: ID does not exist" containerID="8a89c5f9f41c4688dde5f59524224228d08d1e87cbf406ea4fd3bbde237c0174" Jul 29 09:39:57.022784 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:57.022717 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a89c5f9f41c4688dde5f59524224228d08d1e87cbf406ea4fd3bbde237c0174"} err="failed to get container status \"8a89c5f9f41c4688dde5f59524224228d08d1e87cbf406ea4fd3bbde237c0174\": rpc error: code = NotFound desc = could not find container \"8a89c5f9f41c4688dde5f59524224228d08d1e87cbf406ea4fd3bbde237c0174\": container with ID starting with 8a89c5f9f41c4688dde5f59524224228d08d1e87cbf406ea4fd3bbde237c0174 not found: ID does not exist" Jul 29 09:39:57.037939 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:57.037917 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gll82"] Jul 29 09:39:57.040463 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:57.040441 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gll82"] Jul 29 09:39:57.362520 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:39:57.362441 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="319aaefc-deac-41b0-ab8f-1dfded39f59f" path="/var/lib/kubelet/pods/319aaefc-deac-41b0-ab8f-1dfded39f59f/volumes" Jul 29 09:40:01.738242 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:01.738207 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd"] Jul 29 09:40:01.740614 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:01.738684 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="319aaefc-deac-41b0-ab8f-1dfded39f59f" containerName="registry-server" Jul 29 09:40:01.740614 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:01.738698 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="319aaefc-deac-41b0-ab8f-1dfded39f59f" containerName="registry-server" Jul 29 09:40:01.740614 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:01.738775 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="319aaefc-deac-41b0-ab8f-1dfded39f59f" containerName="registry-server" Jul 29 09:40:01.741674 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:01.741659 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" Jul 29 09:40:01.744642 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:01.744618 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-150c68-dockercfg-hjk68\"" Jul 29 09:40:01.744781 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:01.744703 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-150c68\"" Jul 29 09:40:01.753469 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:01.753446 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd"] Jul 29 09:40:01.802417 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:01.802391 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d853b6-915e-4239-89cb-cd7edc753527-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd\" (UID: \"11d853b6-915e-4239-89cb-cd7edc753527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" Jul 29 09:40:01.802565 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:01.802459 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48sgk\" (UniqueName: \"kubernetes.io/projected/11d853b6-915e-4239-89cb-cd7edc753527-kube-api-access-48sgk\") pod \"payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd\" (UID: \"11d853b6-915e-4239-89cb-cd7edc753527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" Jul 29 09:40:01.903290 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:01.903262 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48sgk\" (UniqueName: \"kubernetes.io/projected/11d853b6-915e-4239-89cb-cd7edc753527-kube-api-access-48sgk\") pod \"payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd\" (UID: \"11d853b6-915e-4239-89cb-cd7edc753527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" Jul 29 09:40:01.903467 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:01.903317 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d853b6-915e-4239-89cb-cd7edc753527-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd\" (UID: \"11d853b6-915e-4239-89cb-cd7edc753527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" Jul 29 09:40:01.904007 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:01.903989 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d853b6-915e-4239-89cb-cd7edc753527-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd\" (UID: \"11d853b6-915e-4239-89cb-cd7edc753527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" Jul 29 09:40:01.913915 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:01.913887 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48sgk\" (UniqueName: \"kubernetes.io/projected/11d853b6-915e-4239-89cb-cd7edc753527-kube-api-access-48sgk\") pod \"payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd\" (UID: \"11d853b6-915e-4239-89cb-cd7edc753527\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" Jul 29 09:40:02.052697 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:02.052608 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" Jul 29 09:40:02.188937 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:02.188912 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd"] Jul 29 09:40:02.191218 ip-10-0-141-66 kubenswrapper[2563]: W0729 09:40:02.191190 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d853b6_915e_4239_89cb_cd7edc753527.slice/crio-981048a6cb908e804c71578d4f1415beec13a09f88d810791331321404ee31df WatchSource:0}: Error finding container 981048a6cb908e804c71578d4f1415beec13a09f88d810791331321404ee31df: Status 404 returned error can't find the container with id 981048a6cb908e804c71578d4f1415beec13a09f88d810791331321404ee31df Jul 29 09:40:03.046481 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:03.046427 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" event={"ID":"11d853b6-915e-4239-89cb-cd7edc753527","Type":"ContainerStarted","Data":"981048a6cb908e804c71578d4f1415beec13a09f88d810791331321404ee31df"} Jul 29 09:40:04.053535 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:04.053437 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" event={"ID":"11d853b6-915e-4239-89cb-cd7edc753527","Type":"ContainerStarted","Data":"0d0b256e708b90103cb9a4d36b98bed92f6a81eea24164a42dabbd06c45d5b7a"} Jul 29 09:40:04.053914 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:04.053546 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" Jul 29 09:40:04.070801 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:04.070752 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" podStartSLOduration=1.49933822 podStartE2EDuration="3.07073866s" totalImagesPullingTime="1.57140044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 09:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 09:40:04.069139947 +0000 UTC m=+3117.193240747" watchObservedRunningTime="2026-07-29 09:40:04.07073866 +0000 UTC m=+3117.194839459" Jul 29 09:40:10.414829 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:10.414785 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:40:15.060452 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:15.060411 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" Jul 29 09:40:32.493340 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:32.493260 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd_11d853b6-915e-4239-89cb-cd7edc753527/payload-processing/0.log" Jul 29 09:40:39.816869 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:39.816838 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9"] Jul 29 09:40:39.820646 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:39.820625 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:39.825428 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:39.825403 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b9d569\"/\"default-dockercfg-n2fn6\"" Jul 29 09:40:39.825523 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:39.825440 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b9d569\"/\"openshift-service-ca.crt\"" Jul 29 09:40:39.825523 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:39.825456 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b9d569\"/\"kube-root-ca.crt\"" Jul 29 09:40:39.825523 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:39.825462 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b9d569\"/\"ipp-route-b9d569-kserve-self-signed-certs\"" Jul 29 09:40:39.834783 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:39.834759 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9"] Jul 29 09:40:39.953190 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:39.953149 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-home\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:39.953483 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:39.953223 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-model-cache\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:39.953483 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:39.953268 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-tmp-dir\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:39.953483 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:39.953298 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22dsp\" (UniqueName: \"kubernetes.io/projected/22cf0a82-887c-4329-a4be-0fc9c4715150-kube-api-access-22dsp\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:39.953483 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:39.953380 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-dshm\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:39.953483 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:39.953429 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cf0a82-887c-4329-a4be-0fc9c4715150-tls-certs\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:40.054507 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:40.054464 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-tmp-dir\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:40.054705 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:40.054513 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22dsp\" (UniqueName: \"kubernetes.io/projected/22cf0a82-887c-4329-a4be-0fc9c4715150-kube-api-access-22dsp\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:40.054705 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:40.054567 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-dshm\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:40.054705 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:40.054594 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cf0a82-887c-4329-a4be-0fc9c4715150-tls-certs\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:40.054705 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:40.054632 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-home\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:40.054705 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:40.054693 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-model-cache\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:40.054977 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:40.054957 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-tmp-dir\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:40.055030 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:40.054997 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-model-cache\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:40.055071 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:40.055052 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-home\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:40.056988 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:40.056953 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-dshm\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:40.057238 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:40.057220 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cf0a82-887c-4329-a4be-0fc9c4715150-tls-certs\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:40.064877 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:40.064848 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22dsp\" (UniqueName: \"kubernetes.io/projected/22cf0a82-887c-4329-a4be-0fc9c4715150-kube-api-access-22dsp\") pod \"ipp-route-b9d569-kserve-644dfc64f9-bljs9\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:40.130847 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:40.130749 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:40.268104 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:40.268078 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9"] Jul 29 09:40:40.270268 ip-10-0-141-66 kubenswrapper[2563]: W0729 09:40:40.270243 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22cf0a82_887c_4329_a4be_0fc9c4715150.slice/crio-809d29023dfcbb96642346af19854afa142288ce2358cb46bd7ef3ffaad44c17 WatchSource:0}: Error finding container 809d29023dfcbb96642346af19854afa142288ce2358cb46bd7ef3ffaad44c17: Status 404 returned error can't find the container with id 809d29023dfcbb96642346af19854afa142288ce2358cb46bd7ef3ffaad44c17 Jul 29 09:40:41.235913 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:41.235874 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" event={"ID":"22cf0a82-887c-4329-a4be-0fc9c4715150","Type":"ContainerStarted","Data":"b2b4625c9986c1b3ec5f07a21e3f45c9491153c722ef2e4e40dc5e8a22880331"} Jul 29 09:40:41.236411 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:41.235923 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" event={"ID":"22cf0a82-887c-4329-a4be-0fc9c4715150","Type":"ContainerStarted","Data":"809d29023dfcbb96642346af19854afa142288ce2358cb46bd7ef3ffaad44c17"} Jul 29 09:40:41.255042 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:41.254991 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" podStartSLOduration=2.254976347 podStartE2EDuration="2.254976347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 09:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 09:40:41.253912796 +0000 UTC m=+3154.378013595" watchObservedRunningTime="2026-07-29 09:40:41.254976347 +0000 UTC m=+3154.379077193" Jul 29 09:40:44.118244 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:44.118206 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:40:50.130950 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:50.130911 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:50.131577 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:50.130985 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:50.143236 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:50.143211 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:40:50.290044 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:40:50.290015 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:41:12.238117 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:12.238081 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:41:22.562248 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:22.562221 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd_11d853b6-915e-4239-89cb-cd7edc753527/payload-processing/0.log" Jul 29 09:41:26.851268 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:26.851237 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds"] Jul 29 09:41:26.854785 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:26.854768 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" Jul 29 09:41:26.858835 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:26.858812 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-82acc2-dockercfg-qknt4\"" Jul 29 09:41:26.859226 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:26.859208 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-82acc2\"" Jul 29 09:41:26.867671 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:26.867647 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds"] Jul 29 09:41:26.978807 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:26.978775 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515c51f5-c92f-4e32-a005-ef21e81f1902-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds\" (UID: \"515c51f5-c92f-4e32-a005-ef21e81f1902\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" Jul 29 09:41:26.978961 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:26.978908 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxlx9\" (UniqueName: \"kubernetes.io/projected/515c51f5-c92f-4e32-a005-ef21e81f1902-kube-api-access-rxlx9\") pod \"payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds\" (UID: \"515c51f5-c92f-4e32-a005-ef21e81f1902\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" Jul 29 09:41:27.079592 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:27.079554 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxlx9\" (UniqueName: \"kubernetes.io/projected/515c51f5-c92f-4e32-a005-ef21e81f1902-kube-api-access-rxlx9\") pod \"payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds\" (UID: \"515c51f5-c92f-4e32-a005-ef21e81f1902\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" Jul 29 09:41:27.079771 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:27.079603 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515c51f5-c92f-4e32-a005-ef21e81f1902-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds\" (UID: \"515c51f5-c92f-4e32-a005-ef21e81f1902\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" Jul 29 09:41:27.080254 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:27.080234 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515c51f5-c92f-4e32-a005-ef21e81f1902-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds\" (UID: \"515c51f5-c92f-4e32-a005-ef21e81f1902\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" Jul 29 09:41:27.090118 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:27.090088 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxlx9\" (UniqueName: \"kubernetes.io/projected/515c51f5-c92f-4e32-a005-ef21e81f1902-kube-api-access-rxlx9\") pod \"payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds\" (UID: \"515c51f5-c92f-4e32-a005-ef21e81f1902\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" Jul 29 09:41:27.166589 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:27.166558 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" Jul 29 09:41:27.303847 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:27.303823 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds"] Jul 29 09:41:27.305790 ip-10-0-141-66 kubenswrapper[2563]: W0729 09:41:27.305760 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515c51f5_c92f_4e32_a005_ef21e81f1902.slice/crio-29930ee76e73e66513dc2d93c9f81992faa211ecf3f9de5e5f40d065bd873e8a WatchSource:0}: Error finding container 29930ee76e73e66513dc2d93c9f81992faa211ecf3f9de5e5f40d065bd873e8a: Status 404 returned error can't find the container with id 29930ee76e73e66513dc2d93c9f81992faa211ecf3f9de5e5f40d065bd873e8a Jul 29 09:41:27.467710 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:27.467614 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" event={"ID":"515c51f5-c92f-4e32-a005-ef21e81f1902","Type":"ContainerStarted","Data":"c2a013d83fed6a9d07720ed0678af381dfc518a22371523e11fdca0f06fe192e"} Jul 29 09:41:27.467710 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:27.467680 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" event={"ID":"515c51f5-c92f-4e32-a005-ef21e81f1902","Type":"ContainerStarted","Data":"29930ee76e73e66513dc2d93c9f81992faa211ecf3f9de5e5f40d065bd873e8a"} Jul 29 09:41:27.467903 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:27.467800 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" Jul 29 09:41:27.492830 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:27.492786 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" podStartSLOduration=1.492770404 podStartE2EDuration="1.492770404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 09:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 09:41:27.490573434 +0000 UTC m=+3200.614674243" watchObservedRunningTime="2026-07-29 09:41:27.492770404 +0000 UTC m=+3200.616871226" Jul 29 09:41:30.418149 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:30.418114 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:41:38.474486 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:38.474449 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" Jul 29 09:41:57.050952 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.050918 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds"] Jul 29 09:41:57.051504 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.051200 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" podUID="515c51f5-c92f-4e32-a005-ef21e81f1902" containerName="payload-pre-processing" containerID="cri-o://c2a013d83fed6a9d07720ed0678af381dfc518a22371523e11fdca0f06fe192e" gracePeriod=30 Jul 29 09:41:57.296464 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.296435 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" Jul 29 09:41:57.342046 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.341976 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxlx9\" (UniqueName: \"kubernetes.io/projected/515c51f5-c92f-4e32-a005-ef21e81f1902-kube-api-access-rxlx9\") pod \"515c51f5-c92f-4e32-a005-ef21e81f1902\" (UID: \"515c51f5-c92f-4e32-a005-ef21e81f1902\") " Jul 29 09:41:57.342185 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.342167 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515c51f5-c92f-4e32-a005-ef21e81f1902-plugins-config-volume\") pod \"515c51f5-c92f-4e32-a005-ef21e81f1902\" (UID: \"515c51f5-c92f-4e32-a005-ef21e81f1902\") " Jul 29 09:41:57.342541 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.342517 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515c51f5-c92f-4e32-a005-ef21e81f1902-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "515c51f5-c92f-4e32-a005-ef21e81f1902" (UID: "515c51f5-c92f-4e32-a005-ef21e81f1902"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 09:41:57.344135 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.344111 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515c51f5-c92f-4e32-a005-ef21e81f1902-kube-api-access-rxlx9" (OuterVolumeSpecName: "kube-api-access-rxlx9") pod "515c51f5-c92f-4e32-a005-ef21e81f1902" (UID: "515c51f5-c92f-4e32-a005-ef21e81f1902"). InnerVolumeSpecName "kube-api-access-rxlx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 09:41:57.443268 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.443241 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515c51f5-c92f-4e32-a005-ef21e81f1902-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:41:57.443268 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.443264 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxlx9\" (UniqueName: \"kubernetes.io/projected/515c51f5-c92f-4e32-a005-ef21e81f1902-kube-api-access-rxlx9\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:41:57.615955 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.615870 2563 generic.go:356] "Generic (PLEG): container finished" podID="515c51f5-c92f-4e32-a005-ef21e81f1902" containerID="c2a013d83fed6a9d07720ed0678af381dfc518a22371523e11fdca0f06fe192e" exitCode=0 Jul 29 09:41:57.615955 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.615914 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" Jul 29 09:41:57.616168 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.615915 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" event={"ID":"515c51f5-c92f-4e32-a005-ef21e81f1902","Type":"ContainerDied","Data":"c2a013d83fed6a9d07720ed0678af381dfc518a22371523e11fdca0f06fe192e"} Jul 29 09:41:57.616168 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.616039 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds" event={"ID":"515c51f5-c92f-4e32-a005-ef21e81f1902","Type":"ContainerDied","Data":"29930ee76e73e66513dc2d93c9f81992faa211ecf3f9de5e5f40d065bd873e8a"} Jul 29 09:41:57.616168 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.616067 2563 scope.go:117] "RemoveContainer" containerID="c2a013d83fed6a9d07720ed0678af381dfc518a22371523e11fdca0f06fe192e" Jul 29 09:41:57.630558 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.630538 2563 scope.go:117] "RemoveContainer" containerID="c2a013d83fed6a9d07720ed0678af381dfc518a22371523e11fdca0f06fe192e" Jul 29 09:41:57.630840 ip-10-0-141-66 kubenswrapper[2563]: E0729 09:41:57.630815 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a013d83fed6a9d07720ed0678af381dfc518a22371523e11fdca0f06fe192e\": container with ID starting with c2a013d83fed6a9d07720ed0678af381dfc518a22371523e11fdca0f06fe192e not found: ID does not exist" containerID="c2a013d83fed6a9d07720ed0678af381dfc518a22371523e11fdca0f06fe192e" Jul 29 09:41:57.630939 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.630845 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a013d83fed6a9d07720ed0678af381dfc518a22371523e11fdca0f06fe192e"} err="failed to get container status \"c2a013d83fed6a9d07720ed0678af381dfc518a22371523e11fdca0f06fe192e\": rpc error: code = NotFound desc = could not find container \"c2a013d83fed6a9d07720ed0678af381dfc518a22371523e11fdca0f06fe192e\": container with ID starting with c2a013d83fed6a9d07720ed0678af381dfc518a22371523e11fdca0f06fe192e not found: ID does not exist" Jul 29 09:41:57.637622 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.637601 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds"] Jul 29 09:41:57.641696 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:57.641676 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-82acc2-5bdb84f8d4-vrxds"] Jul 29 09:41:59.362295 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:41:59.362218 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="515c51f5-c92f-4e32-a005-ef21e81f1902" path="/var/lib/kubelet/pods/515c51f5-c92f-4e32-a005-ef21e81f1902/volumes" Jul 29 09:42:01.617061 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:01.617020 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:42:21.927275 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:21.927236 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:42:28.196268 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:28.196238 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9"] Jul 29 09:42:32.913087 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:32.913051 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:42:43.215503 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.215456 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" podUID="22cf0a82-887c-4329-a4be-0fc9c4715150" containerName="main" containerID="cri-o://b2b4625c9986c1b3ec5f07a21e3f45c9491153c722ef2e4e40dc5e8a22880331" gracePeriod=45 Jul 29 09:42:43.467548 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.467486 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:42:43.552109 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.552079 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-model-cache\") pod \"22cf0a82-887c-4329-a4be-0fc9c4715150\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " Jul 29 09:42:43.552280 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.552145 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-dshm\") pod \"22cf0a82-887c-4329-a4be-0fc9c4715150\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " Jul 29 09:42:43.552280 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.552171 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-tmp-dir\") pod \"22cf0a82-887c-4329-a4be-0fc9c4715150\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " Jul 29 09:42:43.552280 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.552194 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-home\") pod \"22cf0a82-887c-4329-a4be-0fc9c4715150\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " Jul 29 09:42:43.552280 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.552230 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22dsp\" (UniqueName: \"kubernetes.io/projected/22cf0a82-887c-4329-a4be-0fc9c4715150-kube-api-access-22dsp\") pod \"22cf0a82-887c-4329-a4be-0fc9c4715150\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " Jul 29 09:42:43.552588 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.552286 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cf0a82-887c-4329-a4be-0fc9c4715150-tls-certs\") pod \"22cf0a82-887c-4329-a4be-0fc9c4715150\" (UID: \"22cf0a82-887c-4329-a4be-0fc9c4715150\") " Jul 29 09:42:43.552588 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.552422 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-model-cache" (OuterVolumeSpecName: "model-cache") pod "22cf0a82-887c-4329-a4be-0fc9c4715150" (UID: "22cf0a82-887c-4329-a4be-0fc9c4715150"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:42:43.552696 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.552639 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22cf0a82-887c-4329-a4be-0fc9c4715150" (UID: "22cf0a82-887c-4329-a4be-0fc9c4715150"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:42:43.552696 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.552646 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-model-cache\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:42:43.552809 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.552718 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-home" (OuterVolumeSpecName: "home") pod "22cf0a82-887c-4329-a4be-0fc9c4715150" (UID: "22cf0a82-887c-4329-a4be-0fc9c4715150"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:42:43.554337 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.554314 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-dshm" (OuterVolumeSpecName: "dshm") pod "22cf0a82-887c-4329-a4be-0fc9c4715150" (UID: "22cf0a82-887c-4329-a4be-0fc9c4715150"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 09:42:43.554573 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.554549 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cf0a82-887c-4329-a4be-0fc9c4715150-kube-api-access-22dsp" (OuterVolumeSpecName: "kube-api-access-22dsp") pod "22cf0a82-887c-4329-a4be-0fc9c4715150" (UID: "22cf0a82-887c-4329-a4be-0fc9c4715150"). InnerVolumeSpecName "kube-api-access-22dsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 09:42:43.554631 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.554574 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cf0a82-887c-4329-a4be-0fc9c4715150-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22cf0a82-887c-4329-a4be-0fc9c4715150" (UID: "22cf0a82-887c-4329-a4be-0fc9c4715150"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 09:42:43.653877 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.653841 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22cf0a82-887c-4329-a4be-0fc9c4715150-tls-certs\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:42:43.653877 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.653870 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-dshm\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:42:43.653877 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.653879 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-tmp-dir\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:42:43.653877 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.653887 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22cf0a82-887c-4329-a4be-0fc9c4715150-home\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:42:43.654152 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.653896 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22dsp\" (UniqueName: \"kubernetes.io/projected/22cf0a82-887c-4329-a4be-0fc9c4715150-kube-api-access-22dsp\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:42:43.846694 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.846601 2563 generic.go:356] "Generic (PLEG): container finished" podID="22cf0a82-887c-4329-a4be-0fc9c4715150" containerID="b2b4625c9986c1b3ec5f07a21e3f45c9491153c722ef2e4e40dc5e8a22880331" exitCode=0 Jul 29 09:42:43.846694 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.846649 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" Jul 29 09:42:43.846879 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.846689 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" event={"ID":"22cf0a82-887c-4329-a4be-0fc9c4715150","Type":"ContainerDied","Data":"b2b4625c9986c1b3ec5f07a21e3f45c9491153c722ef2e4e40dc5e8a22880331"} Jul 29 09:42:43.846879 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.846736 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9" event={"ID":"22cf0a82-887c-4329-a4be-0fc9c4715150","Type":"ContainerDied","Data":"809d29023dfcbb96642346af19854afa142288ce2358cb46bd7ef3ffaad44c17"} Jul 29 09:42:43.846879 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.846753 2563 scope.go:117] "RemoveContainer" containerID="b2b4625c9986c1b3ec5f07a21e3f45c9491153c722ef2e4e40dc5e8a22880331" Jul 29 09:42:43.856512 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.856486 2563 scope.go:117] "RemoveContainer" containerID="b2b4625c9986c1b3ec5f07a21e3f45c9491153c722ef2e4e40dc5e8a22880331" Jul 29 09:42:43.856763 ip-10-0-141-66 kubenswrapper[2563]: E0729 09:42:43.856744 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b4625c9986c1b3ec5f07a21e3f45c9491153c722ef2e4e40dc5e8a22880331\": container with ID starting with b2b4625c9986c1b3ec5f07a21e3f45c9491153c722ef2e4e40dc5e8a22880331 not found: ID does not exist" containerID="b2b4625c9986c1b3ec5f07a21e3f45c9491153c722ef2e4e40dc5e8a22880331" Jul 29 09:42:43.856812 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.856771 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b4625c9986c1b3ec5f07a21e3f45c9491153c722ef2e4e40dc5e8a22880331"} err="failed to get container status \"b2b4625c9986c1b3ec5f07a21e3f45c9491153c722ef2e4e40dc5e8a22880331\": rpc error: code = NotFound desc = could not find container \"b2b4625c9986c1b3ec5f07a21e3f45c9491153c722ef2e4e40dc5e8a22880331\": container with ID starting with b2b4625c9986c1b3ec5f07a21e3f45c9491153c722ef2e4e40dc5e8a22880331 not found: ID does not exist" Jul 29 09:42:43.869462 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.869426 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9"] Jul 29 09:42:43.871448 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:43.871427 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b9d569/ipp-route-b9d569-kserve-644dfc64f9-bljs9"] Jul 29 09:42:45.362419 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:42:45.362386 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22cf0a82-887c-4329-a4be-0fc9c4715150" path="/var/lib/kubelet/pods/22cf0a82-887c-4329-a4be-0fc9c4715150/volumes" Jul 29 09:43:26.248101 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.248067 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97"] Jul 29 09:43:26.248580 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.248434 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22cf0a82-887c-4329-a4be-0fc9c4715150" containerName="main" Jul 29 09:43:26.248580 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.248446 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="22cf0a82-887c-4329-a4be-0fc9c4715150" containerName="main" Jul 29 09:43:26.248580 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.248464 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515c51f5-c92f-4e32-a005-ef21e81f1902" containerName="payload-pre-processing" Jul 29 09:43:26.248580 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.248469 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="515c51f5-c92f-4e32-a005-ef21e81f1902" containerName="payload-pre-processing" Jul 29 09:43:26.248580 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.248530 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="22cf0a82-887c-4329-a4be-0fc9c4715150" containerName="main" Jul 29 09:43:26.248580 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.248543 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="515c51f5-c92f-4e32-a005-ef21e81f1902" containerName="payload-pre-processing" Jul 29 09:43:26.251547 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.251531 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97" Jul 29 09:43:26.254387 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.254356 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 09:43:26.254716 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.254702 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tkm62\"" Jul 29 09:43:26.260760 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.260737 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97"] Jul 29 09:43:26.350262 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.350229 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/836f4acc-1871-4bbb-b0d4-1d44ab58ae34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97\" (UID: \"836f4acc-1871-4bbb-b0d4-1d44ab58ae34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97" Jul 29 09:43:26.451730 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.451697 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/836f4acc-1871-4bbb-b0d4-1d44ab58ae34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97\" (UID: \"836f4acc-1871-4bbb-b0d4-1d44ab58ae34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97" Jul 29 09:43:26.452276 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.452258 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/836f4acc-1871-4bbb-b0d4-1d44ab58ae34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97\" (UID: \"836f4acc-1871-4bbb-b0d4-1d44ab58ae34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97" Jul 29 09:43:26.562722 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.562644 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97" Jul 29 09:43:26.698328 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:26.698306 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97"] Jul 29 09:43:26.700767 ip-10-0-141-66 kubenswrapper[2563]: W0729 09:43:26.700740 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836f4acc_1871_4bbb_b0d4_1d44ab58ae34.slice/crio-149715bcabe4ef1bfceec1c5570da2bb46c05fa348bdebd61324c5b2da7b3bde WatchSource:0}: Error finding container 149715bcabe4ef1bfceec1c5570da2bb46c05fa348bdebd61324c5b2da7b3bde: Status 404 returned error can't find the container with id 149715bcabe4ef1bfceec1c5570da2bb46c05fa348bdebd61324c5b2da7b3bde Jul 29 09:43:27.058554 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:27.058519 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97" event={"ID":"836f4acc-1871-4bbb-b0d4-1d44ab58ae34","Type":"ContainerStarted","Data":"149715bcabe4ef1bfceec1c5570da2bb46c05fa348bdebd61324c5b2da7b3bde"} Jul 29 09:43:29.069617 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:29.069573 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97" event={"ID":"836f4acc-1871-4bbb-b0d4-1d44ab58ae34","Type":"ContainerStarted","Data":"ab4941f9921c570859bdfe0915ac752aae41276e779a3a21129669118439db1e"} Jul 29 09:43:29.117846 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:29.117797 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97" podStartSLOduration=1.112037073 podStartE2EDuration="3.117781505s" totalImagesPullingTime="2.005744432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 09:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 09:43:29.116560377 +0000 UTC m=+3322.240661178" watchObservedRunningTime="2026-07-29 09:43:29.117781505 +0000 UTC m=+3322.241882304" Jul 29 09:43:30.075791 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:30.075755 2563 generic.go:356] "Generic (PLEG): container finished" podID="836f4acc-1871-4bbb-b0d4-1d44ab58ae34" containerID="ab4941f9921c570859bdfe0915ac752aae41276e779a3a21129669118439db1e" exitCode=0 Jul 29 09:43:30.076198 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:30.075849 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97" event={"ID":"836f4acc-1871-4bbb-b0d4-1d44ab58ae34","Type":"ContainerDied","Data":"ab4941f9921c570859bdfe0915ac752aae41276e779a3a21129669118439db1e"} Jul 29 09:43:31.208652 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:31.208626 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97" Jul 29 09:43:31.294316 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:31.294280 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/836f4acc-1871-4bbb-b0d4-1d44ab58ae34-maas-api-service-ca\") pod \"836f4acc-1871-4bbb-b0d4-1d44ab58ae34\" (UID: \"836f4acc-1871-4bbb-b0d4-1d44ab58ae34\") " Jul 29 09:43:31.294673 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:31.294651 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836f4acc-1871-4bbb-b0d4-1d44ab58ae34-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "836f4acc-1871-4bbb-b0d4-1d44ab58ae34" (UID: "836f4acc-1871-4bbb-b0d4-1d44ab58ae34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 09:43:31.395886 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:31.395856 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/836f4acc-1871-4bbb-b0d4-1d44ab58ae34-maas-api-service-ca\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:43:32.086323 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:32.086287 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97" event={"ID":"836f4acc-1871-4bbb-b0d4-1d44ab58ae34","Type":"ContainerDied","Data":"149715bcabe4ef1bfceec1c5570da2bb46c05fa348bdebd61324c5b2da7b3bde"} Jul 29 09:43:32.086323 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:32.086308 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97" Jul 29 09:43:32.086540 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:32.086338 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="149715bcabe4ef1bfceec1c5570da2bb46c05fa348bdebd61324c5b2da7b3bde" Jul 29 09:43:36.881990 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:36.881958 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd"] Jul 29 09:43:36.882490 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:36.882214 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" podUID="11d853b6-915e-4239-89cb-cd7edc753527" containerName="payload-processing" containerID="cri-o://0d0b256e708b90103cb9a4d36b98bed92f6a81eea24164a42dabbd06c45d5b7a" gracePeriod=30 Jul 29 09:43:37.039284 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:37.039247 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97"] Jul 29 09:43:37.041643 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:37.041622 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-150c68-9gc97"] Jul 29 09:43:37.113555 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:37.113521 2563 generic.go:356] "Generic (PLEG): container finished" podID="11d853b6-915e-4239-89cb-cd7edc753527" containerID="0d0b256e708b90103cb9a4d36b98bed92f6a81eea24164a42dabbd06c45d5b7a" exitCode=0 Jul 29 09:43:37.113702 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:37.113597 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" event={"ID":"11d853b6-915e-4239-89cb-cd7edc753527","Type":"ContainerDied","Data":"0d0b256e708b90103cb9a4d36b98bed92f6a81eea24164a42dabbd06c45d5b7a"} Jul 29 09:43:37.138510 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:37.138456 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" Jul 29 09:43:37.248719 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:37.248684 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48sgk\" (UniqueName: \"kubernetes.io/projected/11d853b6-915e-4239-89cb-cd7edc753527-kube-api-access-48sgk\") pod \"11d853b6-915e-4239-89cb-cd7edc753527\" (UID: \"11d853b6-915e-4239-89cb-cd7edc753527\") " Jul 29 09:43:37.248913 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:37.248729 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d853b6-915e-4239-89cb-cd7edc753527-plugins-config-volume\") pod \"11d853b6-915e-4239-89cb-cd7edc753527\" (UID: \"11d853b6-915e-4239-89cb-cd7edc753527\") " Jul 29 09:43:37.249109 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:37.249087 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d853b6-915e-4239-89cb-cd7edc753527-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11d853b6-915e-4239-89cb-cd7edc753527" (UID: "11d853b6-915e-4239-89cb-cd7edc753527"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 09:43:37.250812 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:37.250793 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d853b6-915e-4239-89cb-cd7edc753527-kube-api-access-48sgk" (OuterVolumeSpecName: "kube-api-access-48sgk") pod "11d853b6-915e-4239-89cb-cd7edc753527" (UID: "11d853b6-915e-4239-89cb-cd7edc753527"). InnerVolumeSpecName "kube-api-access-48sgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 09:43:37.350168 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:37.350135 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48sgk\" (UniqueName: \"kubernetes.io/projected/11d853b6-915e-4239-89cb-cd7edc753527-kube-api-access-48sgk\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:43:37.350168 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:37.350164 2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d853b6-915e-4239-89cb-cd7edc753527-plugins-config-volume\") on node \"ip-10-0-141-66.ec2.internal\" DevicePath \"\"" Jul 29 09:43:37.362971 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:37.362939 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="836f4acc-1871-4bbb-b0d4-1d44ab58ae34" path="/var/lib/kubelet/pods/836f4acc-1871-4bbb-b0d4-1d44ab58ae34/volumes" Jul 29 09:43:38.120204 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:38.120173 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" Jul 29 09:43:38.120204 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:38.120196 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd" event={"ID":"11d853b6-915e-4239-89cb-cd7edc753527","Type":"ContainerDied","Data":"981048a6cb908e804c71578d4f1415beec13a09f88d810791331321404ee31df"} Jul 29 09:43:38.120755 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:38.120259 2563 scope.go:117] "RemoveContainer" containerID="0d0b256e708b90103cb9a4d36b98bed92f6a81eea24164a42dabbd06c45d5b7a" Jul 29 09:43:38.138388 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:38.138346 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd"] Jul 29 09:43:38.142847 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:38.142819 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-150c68-654b964bf8-74bpd"] Jul 29 09:43:39.362167 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:39.362132 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11d853b6-915e-4239-89cb-cd7edc753527" path="/var/lib/kubelet/pods/11d853b6-915e-4239-89cb-cd7edc753527/volumes" Jul 29 09:43:40.467084 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:40.467048 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:43:57.525973 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:43:57.525929 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:44:15.218870 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:15.218830 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-824nb"] Jul 29 09:44:19.476919 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:19.476891 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b649d98dc-k4wnc_d1c241f4-c19e-4ab8-987a-e8abfce03ca6/authorino/0.log" Jul 29 09:44:25.314726 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:25.314696 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67764bc7cd-7fsfd_51537977-ebb1-460c-a77d-8292dc399306/copy-manifests/0.log" Jul 29 09:44:25.319922 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:25.319905 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67764bc7cd-7fsfd_51537977-ebb1-460c-a77d-8292dc399306/manager/0.log" Jul 29 09:44:27.398408 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:27.398374 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b649d98dc-k4wnc_d1c241f4-c19e-4ab8-987a-e8abfce03ca6/authorino/0.log" Jul 29 09:44:28.066308 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:28.066277 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-824nb_735bc652-2a81-41bd-aca9-bba5519283da/limitador/0.log" Jul 29 09:44:28.908979 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:28.908948 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d544f58d6-r84vc_8c45b77a-8ead-4131-a581-123128f1df89/kube-auth-proxy/0.log" Jul 29 09:44:29.925198 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:29.925167 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-nb4jv_7ee02e5e-3a4a-4051-9ab9-a0218a0d8de0/main/0.log" Jul 29 09:44:30.034475 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:30.034446 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7rm9l_6793c0d0-ee0a-42dc-a23a-12a9a7afdcb6/main/0.log" Jul 29 09:44:30.254479 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:30.254398 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-mtx5z_66257daf-2ac4-48cc-8b39-aec37967a92a/main/0.log" Jul 29 09:44:38.835744 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:38.835713 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z6kw7_9014122b-1078-4e35-b53c-234e93bbd054/global-pull-secret-syncer/0.log" Jul 29 09:44:38.979084 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:38.979053 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vmwkf_97483690-18fa-4576-801e-4de241e61858/konnectivity-agent/0.log" Jul 29 09:44:39.048861 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:39.048832 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-66.ec2.internal_f6d3bed9801bf6ab9cc93c6f1440b4f0/haproxy/0.log" Jul 29 09:44:44.408342 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:44.408305 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b649d98dc-k4wnc_d1c241f4-c19e-4ab8-987a-e8abfce03ca6/authorino/0.log" Jul 29 09:44:44.869859 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:44.869832 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-824nb_735bc652-2a81-41bd-aca9-bba5519283da/limitador/0.log" Jul 29 09:44:46.541146 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:46.541112 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a62fd3c7-480b-4c33-8a21-b9e566498f9b/alertmanager/0.log" Jul 29 09:44:46.577595 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:46.577566 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a62fd3c7-480b-4c33-8a21-b9e566498f9b/config-reloader/0.log" Jul 29 09:44:46.608720 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:46.608696 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a62fd3c7-480b-4c33-8a21-b9e566498f9b/kube-rbac-proxy-web/0.log" Jul 29 09:44:46.644044 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:46.644020 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a62fd3c7-480b-4c33-8a21-b9e566498f9b/kube-rbac-proxy/0.log" Jul 29 09:44:46.669606 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:46.669581 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a62fd3c7-480b-4c33-8a21-b9e566498f9b/kube-rbac-proxy-metric/0.log" Jul 29 09:44:46.693211 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:46.693180 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a62fd3c7-480b-4c33-8a21-b9e566498f9b/prom-label-proxy/0.log" Jul 29 09:44:46.717057 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:46.717037 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a62fd3c7-480b-4c33-8a21-b9e566498f9b/init-config-reloader/0.log" Jul 29 09:44:46.808848 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:46.808766 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-9nc84_a44c7b97-b8d1-479c-868d-5b59ac55b6b7/kube-state-metrics/0.log" Jul 29 09:44:46.828430 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:46.828404 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-9nc84_a44c7b97-b8d1-479c-868d-5b59ac55b6b7/kube-rbac-proxy-main/0.log" Jul 29 09:44:46.851912 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:46.851892 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-9nc84_a44c7b97-b8d1-479c-868d-5b59ac55b6b7/kube-rbac-proxy-self/0.log" Jul 29 09:44:46.880849 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:46.880828 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-9b4dd9b9c-cxdrg_ecaff838-b293-4dfb-96b0-aeb81daf695b/metrics-server/0.log" Jul 29 09:44:46.905559 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:46.905540 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-z4dvx_baa0986b-a86d-4acf-a9a7-0590bf571e1b/monitoring-plugin/0.log" Jul 29 09:44:47.088779 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:47.088696 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jcvqx_9ee3dae8-2df4-4b54-be97-532bebe98369/node-exporter/0.log" Jul 29 09:44:47.115020 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:47.114994 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jcvqx_9ee3dae8-2df4-4b54-be97-532bebe98369/kube-rbac-proxy/0.log" Jul 29 09:44:47.136778 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:47.136752 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jcvqx_9ee3dae8-2df4-4b54-be97-532bebe98369/init-textfile/0.log" Jul 29 09:44:47.397976 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:47.397890 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-l5xff_f29ccd07-74c4-4c24-ab86-e1df11e9184b/prometheus-operator/0.log" Jul 29 09:44:47.414587 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:47.414556 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-l5xff_f29ccd07-74c4-4c24-ab86-e1df11e9184b/kube-rbac-proxy/0.log" Jul 29 09:44:47.437260 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:47.437227 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-8wcw7_d88f07f1-d4c3-40fd-91d2-76ab5b41c546/prometheus-operator-admission-webhook/0.log" Jul 29 09:44:48.917767 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:48.917741 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-qks4q_41e45e49-9c61-43f9-86a8-1a46d12a4544/networking-console-plugin/0.log" Jul 29 09:44:48.999208 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:48.999177 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj"] Jul 29 09:44:48.999533 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:48.999521 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="836f4acc-1871-4bbb-b0d4-1d44ab58ae34" containerName="revoke-keys" Jul 29 09:44:48.999585 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:48.999535 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="836f4acc-1871-4bbb-b0d4-1d44ab58ae34" containerName="revoke-keys" Jul 29 09:44:48.999585 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:48.999554 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d853b6-915e-4239-89cb-cd7edc753527" containerName="payload-processing" Jul 29 09:44:48.999585 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:48.999559 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d853b6-915e-4239-89cb-cd7edc753527" containerName="payload-processing" Jul 29 09:44:48.999684 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:48.999612 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d853b6-915e-4239-89cb-cd7edc753527" containerName="payload-processing" Jul 29 09:44:48.999684 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:48.999625 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="836f4acc-1871-4bbb-b0d4-1d44ab58ae34" containerName="revoke-keys" Jul 29 09:44:49.002752 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.002736 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.005701 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.005680 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dcgtc\"/\"openshift-service-ca.crt\"" Jul 29 09:44:49.006904 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.006888 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dcgtc\"/\"default-dockercfg-lkspl\"" Jul 29 09:44:49.006996 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.006938 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dcgtc\"/\"kube-root-ca.crt\"" Jul 29 09:44:49.013635 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.013614 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj"] Jul 29 09:44:49.195632 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.195542 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpgcz\" (UniqueName: \"kubernetes.io/projected/2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4-kube-api-access-vpgcz\") pod \"perf-node-gather-daemonset-jl4kj\" (UID: \"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4\") " pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.195632 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.195603 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4-sys\") pod \"perf-node-gather-daemonset-jl4kj\" (UID: \"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4\") " pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.195861 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.195711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4-lib-modules\") pod \"perf-node-gather-daemonset-jl4kj\" (UID: \"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4\") " pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.195861 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.195755 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4-proc\") pod \"perf-node-gather-daemonset-jl4kj\" (UID: \"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4\") " pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.195861 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.195820 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4-podres\") pod \"perf-node-gather-daemonset-jl4kj\" (UID: \"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4\") " pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.297178 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.297129 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4-sys\") pod \"perf-node-gather-daemonset-jl4kj\" (UID: \"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4\") " pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.297178 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.297194 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4-lib-modules\") pod \"perf-node-gather-daemonset-jl4kj\" (UID: \"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4\") " pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.297468 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.297215 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4-proc\") pod \"perf-node-gather-daemonset-jl4kj\" (UID: \"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4\") " pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.297468 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.297235 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4-podres\") pod \"perf-node-gather-daemonset-jl4kj\" (UID: \"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4\") " pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.297468 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.297252 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4-sys\") pod \"perf-node-gather-daemonset-jl4kj\" (UID: \"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4\") " pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.297468 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.297272 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpgcz\" (UniqueName: \"kubernetes.io/projected/2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4-kube-api-access-vpgcz\") pod \"perf-node-gather-daemonset-jl4kj\" (UID: \"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4\") " pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.297468 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.297339 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4-proc\") pod \"perf-node-gather-daemonset-jl4kj\" (UID: \"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4\") " pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.297468 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.297352 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4-podres\") pod \"perf-node-gather-daemonset-jl4kj\" (UID: \"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4\") " pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.297468 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.297410 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4-lib-modules\") pod \"perf-node-gather-daemonset-jl4kj\" (UID: \"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4\") " pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.306040 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.306017 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpgcz\" (UniqueName: \"kubernetes.io/projected/2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4-kube-api-access-vpgcz\") pod \"perf-node-gather-daemonset-jl4kj\" (UID: \"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4\") " pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.312998 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.312979 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:49.442046 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.442023 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj"] Jul 29 09:44:49.444458 ip-10-0-141-66 kubenswrapper[2563]: W0729 09:44:49.444421 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2d0d3ae4_bc5a_4b72_818d_8a13af2b83b4.slice/crio-96e53c37ff44ca584c62bebf61e88f4f81b96efe62e3f30e9b5854a334c97148 WatchSource:0}: Error finding container 96e53c37ff44ca584c62bebf61e88f4f81b96efe62e3f30e9b5854a334c97148: Status 404 returned error can't find the container with id 96e53c37ff44ca584c62bebf61e88f4f81b96efe62e3f30e9b5854a334c97148 Jul 29 09:44:49.468651 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:49.468624 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" event={"ID":"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4","Type":"ContainerStarted","Data":"96e53c37ff44ca584c62bebf61e88f4f81b96efe62e3f30e9b5854a334c97148"} Jul 29 09:44:50.178733 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:50.178697 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-sn9p9_a96b2499-7e0c-4f91-8954-b33007b6753c/download-server/0.log" Jul 29 09:44:50.474820 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:50.474723 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" event={"ID":"2d0d3ae4-bc5a-4b72-818d-8a13af2b83b4","Type":"ContainerStarted","Data":"4c131a7060d9ec2a123973e90e4187e9d990c9b0326f5b6b3f50f8cf6fd23858"} Jul 29 09:44:50.474820 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:50.474785 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:50.495247 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:50.495201 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" podStartSLOduration=2.4951847 podStartE2EDuration="2.4951847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 09:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 09:44:50.495056405 +0000 UTC m=+3403.619157204" watchObservedRunningTime="2026-07-29 09:44:50.4951847 +0000 UTC m=+3403.619285498" Jul 29 09:44:51.889652 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:51.889617 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7ff2z_bc60cc72-c068-4d53-a2ad-4141c76472a5/dns/0.log" Jul 29 09:44:51.914580 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:51.914548 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7ff2z_bc60cc72-c068-4d53-a2ad-4141c76472a5/kube-rbac-proxy/0.log" Jul 29 09:44:52.061859 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:52.061816 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hmb59_86239d8c-0cc4-4be4-b21a-e31d5139934c/dns-node-resolver/0.log" Jul 29 09:44:52.703703 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:52.703673 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cpx2c_1422c6a2-0287-4e57-8379-b18d6fcef17e/node-ca/0.log" Jul 29 09:44:53.984538 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:53.984509 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d544f58d6-r84vc_8c45b77a-8ead-4131-a581-123128f1df89/kube-auth-proxy/0.log" Jul 29 09:44:55.018574 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:55.018541 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4pvfb_9ffc23fe-8cc5-4d63-bc98-fba3c171561b/serve-healthcheck-canary/0.log" Jul 29 09:44:55.732828 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:55.732799 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nfxfj_6c3c1c04-de29-4b84-bbb3-f6e5107979dd/kube-rbac-proxy/0.log" Jul 29 09:44:55.754697 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:55.754676 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nfxfj_6c3c1c04-de29-4b84-bbb3-f6e5107979dd/exporter/0.log" Jul 29 09:44:55.778385 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:55.778338 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nfxfj_6c3c1c04-de29-4b84-bbb3-f6e5107979dd/extractor/0.log" Jul 29 09:44:56.489401 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:56.489377 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dcgtc/perf-node-gather-daemonset-jl4kj" Jul 29 09:44:59.115473 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:59.115448 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67764bc7cd-7fsfd_51537977-ebb1-460c-a77d-8292dc399306/manager/0.log" Jul 29 09:44:59.139199 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:59.139177 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67764bc7cd-7fsfd_51537977-ebb1-460c-a77d-8292dc399306/copy-manifests/0.log" Jul 29 09:44:59.955465 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:44:59.955436 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d544f58d6-r84vc_8c45b77a-8ead-4131-a581-123128f1df89/kube-auth-proxy/0.log" Jul 29 09:45:00.417294 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:45:00.417267 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-848779956-4vchn_7de74cd6-0148-4fcb-b882-8ebb4abd4c01/manager/0.log" Jul 29 09:45:01.888521 ip-10-0-141-66 kubenswrapper[2563]: I0729 09:45:01.888419 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d544f58d6-r84vc_8c45b77a-8ead-4131-a581-123128f1df89/kube-auth-proxy/0.log"